Presentation on theme: "M&E for Accountability & Global Progress Tracking"— Presentation transcript:
1M&E for Accountability & Global Progress Tracking Monitoring, Operations Research and Evaluation TeamUNAIDS/GenevaApril 2007
2ORIENTATION 2008 UNGASS Country Progress Report Benefits of UNGASS for M&E strengtheningUNGASS GuidelinesPurpose and useCore indicatorsData sources & data vettingReporting process
3UNGASS Reporting Timetable INTRODUCTIONWhere does UNGASS fit in M&E system?UNGASS Reporting TimetableUse of 2006 UNGASS dataGlobal useCountry use
4WHERE DOES UNGASS FIT IN A COMPREHENSIVE M&E SYSTEM? Benefits of UNGASS reporting for M&E strengthening:Brings partners together around a core set of indicatorsHelps identify data gaps & data quality issuesStimulates the establishment of a centralised database of HIV indicator dataProvides a de facto bi-annual report on the epidemic and the response
5WHERE DOES UNGASS FIT IN A COMPREHENSIVE M&E SYSTEM? ProgramImprovementShareData withPartnersReporting/Accountability[UNGASS, other]
6UNGASS REPORTING TIMETABLE REPORT REQUIRED?2003Yes2004No2005Interim report from a subset of countries20062007200820092010
7GLOBAL USE OF 2006 UNGASS DATA Global Progress ReportReport to the Secretary General (as required in the Declaration of Commitment)Used for tracking the Millennium Development Goals
8COUNTRY USE OF 2006 UNGASS DATA 1. National report on the epidemic and responseinforms programmatic decisionsguides reviews of the National Strategic Planinforms resource mobilisation efforts (e.g. GFATM proposal development)is used as an advocacy toolis used to report progress on Declaration of Commitment to Senior Government Officials
9COUNTRY USE OF 2006 UNGASS DATA 2. M&E system strengtheningconvenes relevant partners to address a particular M&E needidentifies data gapsassists in planning data collection effortsidentifies data weaknesseshelps mobilise resources for system strengtheningstimulates evaluation questions
10LESSONS LEARNED FROM 2006 UNGASS REPORTING Challenges in 2006 reporting2006 reporting from MENALessons learned from 2006 reportingUNAIDS/Geneva-levelCountry-level
11CHALLENGES IN 2006 REPORTING 137 countries submitted a Country Progress Report10% reported on all indicators relevant to epidemic51% used CRIS to report indicator dataSome basic data quality issues includingincorrect indicator or incorrect indicator calculationincomplete data (e.g. only 20% of indicators reported included the required disaggregation)illogical valuesdiscrepancies between data in CRIS and Report
12Percent Range and Median of Indicators Reported by Countries by Region - UNGASS 2006 ReportHow may indicators did countries report within each region? This slide shows the range of percent reporting, i.e. if countries reported all indicators = 100%; if they reported none=0%. The median refers to the number of indicators 50% of countries reported. For example, 50% of developed countries did not report any indicators, whereas in the Caribbean, 50% of countries reported more than 80% of the relevant indicators.The question is whether a low level of reporting is due to lack of data or something else?
13IMPORTANCE OF DISAGGREGATED DATA Disaggregated data is very important for understanding the epidemic and for program development.For example, to answer the question “Are programs reaching all members of target population”?For example, looking at safe practices in IDU, it is clear that for most countries, except Moldova and Macedonia, more women adopt safer practices than men. From a program perspective, this is important to know as messages for men may need to be tailored differently than for women. Without disaggregated data this information would not be evident.
14LESSONS LEARNED FROM 2006 REPORTING UNAIDS/Geneva-level The following inputs are paramount for improvingthe quality of the data obtained from countries:Revise UNGASS indicators, where necessaryProvide clear and specific GuidelinesProvide user-friendly, flexible CRIS softwareProvide training on UNGASS indicators & reporting processProvide technical support, where needed
15LESSONS LEARNED FROM 2006 REPORTING Country-level Ensure correct understanding of indicator definition/operandsReport all required indicatorsIdentify and document data sourcesProvide all data and information needed for each indicatorCross-validate data (comparison with previous years, comparison with alternate sources, etc.)Use CRIS for indicator reportingDouble-check for data entry errorsSubmit timely
16LESSONS LEARNED FROM 2006 REPORTING LESSONS LEARNED FROM 2006 REPORTING Country-levelLESSONS LEARNED FROM 2006 REPORTINGAddress data gaps earlySeek to enhance data qualityFollow all the steps in the reporting processDouble-check accuracy and obtain consensus on all indicator values at country level before submission
17ORIENTATION ON THE UNGASS GUIDELINES PurposeTarget audienceContentsPrinciples and process of Guidelines revisionUse of Guidelines
19PURPOSE Provide essential information on construction of core indicators for reportingEnsure transparency of the reporting processin-countryEnsure consistency of information across countries for global progress analysisThe primary purpose of the UNGASS guidelines is to provide the key constituents, who are actively involved in a country’s response to HIV and AIDS, with essential information on the core indicators that measure the effectiveness of the national response.The Guidelines also help to ensure transparency of the in-country reporting process in terms of who was involved and how the information was obtained.In addition, they help to ensure consistency of the information obtained from different countries so that a global analysis can be conducted of the progress made towards achieving the targets of the Declaration of Commitment.
20TARGET AUDIENCES Government sector Nongovernmental sector including the National AIDS Committee, Ministries (e.g. Ministry of Health, Ministry of Finance, Ministry of Education)Nongovernmental sectorincluding nongovernmental organizations, faith-based organizations, people living with HIV, private sectorUN co-sponsors and donor agenciesAlthough it was governments that initially endorsed the Declaration of Commitment, the document’s vision extends far beyond the governmental sector—to private sector and labour groups, faith-based organizations, nongovernmental organizations and other civil-society entities, including organizations of people living with HIV
21CONTENTS OVERVIEW Introduction p. 9-13 Implementation at National Level pCore Indicators pAppendices p
22APPENDICES 1 - 9 Changes to Core Indicators Country Progress Report templateMethodology used for the coverage surveyConsultation/preparation process for the Country Progress ReportReporting schedule for Core IndicatorsNational Funding Matrix 2007National Composite Policy Index (NCPI) 2007Sample check-list for Country Progress ReportSelected bibliographyThe Appendices are essential to developing the Country Progress Report and countries are strongly encouraged to read them and use them.
23PRINCIPLES OF GUIDELINES REVISION Consensus-driven processTransparencyConsistencyComprehensivenessQualityIntegration within the National M&E systemFeasibility
24PROCESS OF GUIDELINES REVISION Process of Guidelines Revision Debriefing process2006 Indicator performanceM&E Reference Group reviewTechnical working groups for each indicator
25HOW TO USE THE GUIDELINES Familiarize yourself with the current Guidelines & the changes since the previous roundDisseminate the Guidelines and use them to develop a collaborative in-country processEnsure that the correct definitions are used for construction of the indicators & refer to Further Information, as neededUse all AppendicesAsk for clarification / assistance, as needed
26CORE INDICATORS Overview of Core UNGASS Indicators Which UNGASS indicators to reportChanges since the 2005 UNGASS GuidelinesCore UNGASS Indicators
27WHAT ARE CORE UNGASS INDICATORS? Quantitative variables which provide simple andreliable ways of measuring progress towardsachieving the Declaration of Commitment onHIV/AIDS
28CORE UNGASS INDICATORS National Indicators (N=25)Four categories:1. National commitment and action2. National programmes3. Knowledge and behaviour4. ImpactGlobal Indicators (N=4)There are 2 types of UNGASS indicators: national and global indicators.1) The national indicators are the indicators each country will have to collect data for and we will focus on these in this presentation.2) The global indicators are designed to provide information on levels of international support for key elements of the global response and to identify trends in that support. Although National AIDS Committees or their equivalents have no direct role in reporting on the core global indicators, there is a similar process for collecting, analysing and reporting data, which is handled directly by UNAIDS and its partners.The core UNGASS indicators consist of 25 national indicators and 4 global indicators.
29CORE UNGASS INDICATORS: NATIONAL INDICATORS Four categories:1. National commitment and action2. National programmes3. Knowledge and behaviour4. ImpactNational level UNGASS indicators are divided into 4 categories:The indicators on National commitment and action focus on policy and the strategic and financial inputs for the prevention of the spread of HIV infection, the provision of care and support for people who are infected and the mitigation of the social and economic consequences of high levels of morbidity and mortality due to AIDS. They also capture programme outputs, coverage and outcomes; for example, in preventing the transmission of HIV from mother to child, in providing treatment with antiretroviral therapy for those in need, and services for orphans and vulnerable children.2) The national programme indicators cover testing, prevention and treatment.3) The indicators on National knowledge and behaviour cover a range of specific knowledge and behavioural outcomes, including: accurate knowledge about HIV transmission, age at first sex, sexual behaviours, and school attendance among orphans.4) The indicators for National-level programme impact, such as percentage of young people infected with HIV, focus on the extent to which national programme activities have succeeded in reducing rates of HIV infection and its associated morbidity and mortality.
30CORE UNGASS INDICATORS: NATIONAL INDICATORS National indicators are important for two reasons:They are used to evaluate the effectiveness of the national response2. They are used to provide information on regional and global trendsThe national indicators are important for 2 reasons:The national indicators can help individual countries evaluate the effectiveness of their national response, which reinforces the value of including these indicators in national monitoring and evaluation frameworksThe second reason is when data from multiple countries are analysed collectively, the indicators can provide critical information on the effectiveness of the response at regional and global levels while simultaneously supplying countries with comparative insights into the efforts of other national-level responses
31Millennium Development Goals (MDG) MILLENIUM DEVELOPMENT GOALS (MDGs)Millennium Development Goals (MDG)There are 8 MDGs:Goal 6: Combat HIV/AIDS, Malaria and other diseasesTarget: Have halted by 2015 and begun to reverse the spread of HIV/AIDSUNGASS indicators are used to monitor progressThere are 8 MDGs. Goal 6 relates to HIV/AIDS: “Combat HIV/AIDS, Malaria and other disease”. There are two targets under this goal, the one relating to HIV/AIDS is: Have halted by 2015 and begun to reverse the spread of HIV/AIDS.
32UNGASS Indicators to monitor MDG-6 12) Current school attendanceamong orphans andamong non-orphans aged 10-1417) Percentage of women and menaged 15–49 who had more thanone sexual partner in the past12 months who report the useof a condom during theirlast sexual intercourseFour of the national indicators are also13) Percentage of youngwomen and men aged 15-24who both correctly identify waysof preventing the sexual transmissionof HIV and who reject majormisconceptions aboutHIV transmission22) Percentage of youngwomen and men aged15–24 who are HIV infected4 of the national indicators are also Millennium Development Goal Indicators.These four indicators relate to school attendance among orphans ,knowledge among young people about HIV, condom use, and the percentage of young people who are infected with HIV.
33Global Indicators Amount of bilateral and Amount of public funds for multilateral financial flows(commitments and disbursements)for the benefit of low- andmiddle-income countriesAmount of public funds forresearch and development ofpreventive HIV vaccinesand microbicidesPercentage of transnationalcompanies that are presentin developing countries andthat have workplace HIVpolicies and programmes4) Percentage of internationalorganizations which haveworkplace HIV policies andprogrammesThe 4 global indicators are collected, analyzed and calculated at UNAIDS and its partners. They are- The amount of bilateral and multilateral financial flows (commitments and disbursements) for the benefit of low and middle income countries.- The amount of public funds for Research and Development of preventive HIV vaccines and microbicides.-The percentage of transnational companies which are present in developing countries and which have HIV/AIDS workplace policies and programmes.- Percentage of international organizations which have workplace policies and programmes.
34Which UNGASS indicators should countries report? No distinction between Generalized Epidemic Indicator Set & Concentrated /Low-prevalence Epidemic Indicator SetMost national indicators are applicable for all countriesCountries are expected to "know their epidemic"How are countries supposed to choose the national indicators?In the previous round of UNGASS reporting, indicators were grouped into two categories according to the nature of the epidemic. The distinction between a generalized epidemic indicator set and a concentrated or low-prevalence epidemic set has not been made for this round of reporting. This is due to the fact that epidemics do not fit neatly into simple dichotomous classifications.In 2005, for example, some countries with a generalized epidemic and a relatively high prevalence recognised that sub-epidemics in most-at-risk populations were important in their epidemic and that prevention programmes for these populations were an essential element of their national response. Rightly, these countries chose to report on the indicators for these populations in addition to the generalised indicator set.Most of the national indicators are applicable for all countriesFor this round of reporting, countries are expected to consider each indicator in light of the individual dynamics of their epidemic.That means that each country is expected to “know their epidemic”.
35Which UNGASS indicators should countries report? Which UNGASS indicators should countries report? (continued)Which UNGASS indicators should countries report?When countries choose not to report on aparticular indicator, an explanation needs to beprovided:Either,The indicator is not applicable to the epidemicThe indicator is applicable, but no data is availableWhen countries choose not to report on a particular indicator, they are asked to provide an explanation as to why they chose not to report. This will allow for an analysis that differentiates between an absence of data, and the inapplicability of particular indicators to particular country situations. In other words, all the indicators needs to be addressed either by reporting or by an explanation of why the country is not reporting on it.
36Which UNGASS indicators should countries report? (continued) The indicators have to be constructed including therequired disaggregations, as outlined in thecurrent UNGASS guidelinesThe indicators for UNGASS reporting have to be constructed as outline in the UNGASS guidelines. The data should also be disaggregated when possible, for example by sex and age.
37Changes since the 2005 UNGASS Guidelines Why changes?To improve the comprehensiveness and quality of dataWhat changes?See Appendix 1There has been changes after the last round of reporting in In order to improve the comprehensiveness and quality of data to be submitted for the 2008 Global Progress Report, refinements were made to the 2005 UNGASS indicators and accompanying guidelines. Refinements were based on (1) input received from a variety of partners through an extensive debriefing process, (2) an analysis of indicator performance in the 2005 reporting round and (3) new programmatic developments. These changes are listed in detail in Appendix 1 in the guidelines.
38Changes since the 2005 UNGASS Guidelines (continued) Principles used in revisionschanges are based oninput received from partnersan analysis of indicator performance in the 2005 reporting roundnew programmatic developmentsevery effort was taken to minimize changesevery effort was taken to ensure that most countries would be able to collect the data or obtain it from already existing data sourcesIn all instances, every effort was taken to minimize changes, since clearly it is the consistency of indicators over time that allows for the assessment of trends and progress. Additionally, extra effort was taken to ensure that most countries would be able to collect the data required for the construction of the indicator, or obtain it from existing sources.
39Changes since the 2005 UNGASS Guidelines (continued) Indicators added:6) Percentage estimated HIV-positive incident TB cases that received treatment for TB and HIV7) Percentage of women and men aged who received an HIV test in the last 12 months and who know their resultsTwo indicators have been added for this round of reporting. The two are the percentage estimated HIV positive incident TB cases that received treatment for TB and HIV, and the percentage of women and men aged who received an HIV test in the last 12 months and who know their results. These two are added as a because of their programmatic significance.
40Changes since the 2005 UNGASS Guidelines (continued) Indicators removed:Percentage of large enterprises/companies which have HIV/AIDS workplace policies and programmesPercentage of women and men with sexually transmitted infections at health care facilities who are appropriately diagnosed, treated and counselledTwo of the national indicators have been removed from the UNGASS set, namely:- Percentage of large enterprises/companies which have HIV/AIDS workplace policies and programmes; and,- Percentage of women and men with sexually transmitted infections at health care facilities who are appropriately diagnosed, treated and counselledWorkplace issues have been incorporated into the NCPIThe indicator on sexually transmitted infections (STIs) was removed because of difficulties in getting reliable data on this indicator in the previous round of reporting, not because it is not important anymore. When a new reliable indicator on STIs is developed, it will be added to the national UNGASS indicator set.
41Changes since the 2005 UNGASS Guidelines (continued) Indicator definition changed1) AIDS spending3) Blood Safety5) Prevention of Mother-to-Child TransmissionMost-at-risk Populations: Prevention Programmes11) Life Skills-based HIV Education in Schools16) Higher-risk Sex17) Condom Use During Higher-risk SexSome of the indicators have been revised since the 2005 round of reporting. In some cases, there were relatively minor revisions to the wording of the guidance in order to provide greater clarity. In other cases, the indicators were revised based on new programme developments, response rates to the indicator and the quality of data reported. In all instances every effort was taken to harmonise any changes made with those indicators currently in use by national governments and other development partners. The consistency of indicators over time allows assessment of trends and of progress. The list of changes to the indicators is provided in Appendix 1 in the guidelines.
42Changes since the 2005 UNGASS Guidelines (continued) Age range expanded4) HIV Treatment: Antiretroviral TherapyComposite indicator divided into its components20) Injecting Drug Users: Condom Use21) Injecting Drug Users: Safe Injecting PracticesAdded questions2) National Composite Policy Index
43CORE UNGASS INDICATORS: NATIONAL INDICATORS National Commitment and ActionDomestic and International AIDS spendingby categories and financing sources2) National Composite Policy Index (NCPI)Questionnaire divided into two sections:Part A(Government)Strategic plan; Political support;Prevention; Treatment, care &support; Monitoring &EvaluationPart B(Non-government)Human rights; Civil societyinvolvement; Prevention;Treatment, care & supportThe two first national indicators provide information on National Commitment and Action by measuring each country’s AIDS spendings by financing source and the National CompositePolicy Index (NCPI). The NCPI is designed to assess progress in the developmentand implementation of national HIV/AIDS policies and strategies. It is a questionnaire that isdivided into two parts:Part A to be administered to government officials.And it covers five areas:Strategic planPolitical supportPreventionTreatment, care and supportMonitoring and evaluationPart B to be administered to representatives from nongovernmental organizations, bilateral agencies, and UN organizations.Part B covers four areas:Human rightsCivil society involvementThe overall responsibility for collating and submitting the information requested in the NCPI lies with the National Governments, through officials from the National AIDS Committee (NAC) (or equivalent) with support from UNAIDS and other partners. It is important that there is adequate representation from relevant organizations and civil society in completing the questionnaire. Appendix 7 in the guidelines provides the template for the NCPI and also gives detailed instructions on how to gather data needed for the index.
44CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes3) Percentage of donated blood units screened forHIV in a quality-assured mannerPercentage of adults and children with advancedHIV infection receiving antiretroviral therapy5) Percentage of HIV-positive pregnant women who receivedantiretrovirals to reduce the risk of mother-to-child transmissionThe National programme indicators consist of 9 indicators on testing, prevention and treatment.3) Percentage of donated blood units screened for HIV in a quality-assured manner4) Percentage of women and men with advanced HIV infection receiving antiretroviral therapy5) Percentage of HIV-positive pregnant women who received antiretrovirals to reduce the risk of mother-to-child transmission6) Percentage estimated HIV positive incident TB cases that received treatment for TB and HIV. This indicator is one of two indicators that were added.Percentage estimated HIV-positive incidentTB cases that received treatment for TB and HIV
45CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes (continued)Percentage of women and men aged who receivedan HIV test in the last 12 months and who know their resultsPercentage of most-at-risk populations who received anHIV test in the last 12 months and who know their results7) The second indicator that is new as an UNGASS indicator is the percentage of women and men aged who received an HIV test in the last 12 months and who know their results.Further of the National Programme Indicators are:8) Percentage of (most-at-risk populations) who received an HIV test in the last 12 months and who know their results*Please note that the most at risk populations should be calculated separately for each of the most at risk populations. Most at risk populations are injecting drug users, men who have sex with men, sex workers and clients of sex workers.9) Percentage of (most-at-risk populations) reached by prevention programmes10) Percentage of orphaned and vulnerable children whose households received free basic external support in caring for the child11) Percentage of schools that provided life-skills based HIV/AIDS education within the last academic yearPercentage of most-at-risk populations reachedwith HIV prevention programmes
46CORE UNGASS INDICATORS: NATIONAL INDICATORS National Programmes (continued)10) Percentage of orphaned and vulnerable children aged 0-17 whosehouseholds received free basic external support in caring for the child11) Percentage of schools that provided life-skills basedHIV education within the last academic year7) The second indicator that is new as an UNGASS indicator is the percentage of women and men aged who received an HIV test in the last 12 months and who know their results.Further of the National Programme Indicators are:8) Percentage of (most-at-risk populations) who received an HIV test in the last 12 months and who know their results*Please note that the most at risk populations are put in brackets because they should be calculated separately for each of the most at risk populations. Most at risk populations are injecting drug users, men who have sex with men, sex workers and clients of sex workers.9) Percentage of (most-at-risk populations) reached by prevention programmes10) Percentage of orphaned and vulnerable children whose households received free basic external support in caring for the child11) Percentage of schools that provided life-skills based HIV/AIDS education within the last academic year
47CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour12) Current school attendance among orphans and non-orphans aged 10–14*13) Percentage of young women and men aged 15–24 who both correctlyidentify ways of preventing the sexual transmission of HIV andwho reject major misconceptions about HIV transmission*14) Percentage of most-at-risk populations who both correctly identify ways ofpreventing the sexual transmission of HIV and who rejectmajor misconceptions about HIV transmissionThe Knowledge and Behaviour indicators consists of 10 indicators assessing the level of knowledge and behaviour in regards of HIV/AIDS transmission and protection. They consist of:12) Current school attendance among orphans and non-orphans aged 10–14.13) Percentage of young women and men aged 15–24 who both correctlyidentify ways of preventing the sexual transmission of HIV andwho reject major misconceptions about HIV transmission14) Percentage of (most-at-risk populations) who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission.15) Percentage of young women and men aged who have had sex before the age of 1516) Percentage of adults aged 15–49 who have had sex with more than one partner in the last 12 months.*Millennium Development Goals indicator
48CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour (continued)15) Percentage of young women and menaged who have had sexual intercourse before the age of 1516) Percentage of women and men aged 15–49 who havehad sexual intercourse with more than one partner in the last 12 months17) Percentage of women and men aged 15–49 who had more thanone sexual partner in the past 12 months reporting the use of a condomduring their last sexual intercourse*The Knowledge and Behaviour indicators consists of 10 indicators assessing the level of knowledge and behaviour in regards of HIV/AIDS transmission and protection. They consist of:12) Current school attendance among orphans and non-orphans aged 10–14.13) Percentage of young women and men aged 15–24 who both correctlyidentify ways of preventing the sexual transmission of HIV andwho reject major misconceptions about HIV transmission14) Percentage of (most-at-risk populations) who both correctly identify ways of preventing the sexual transmission of HIV and who reject major misconceptions about HIV transmission.15) Percentage of young women and men aged who have had sex before the age of 1516) Percentage of adults aged 15–49 who have had sex with more than one partner in the last 12 months.*Millennium Development Goals indicator
49CORE UNGASS INDICATORS: NATIONAL INDICATORS Knowledge and Behaviour (continued)18) Percentage of female and male sex workersreporting the use of a condom with their most recent client19) Percentage of men reporting the use of a condom thelast time they had anal sex with a male partner20) Percentage of injecting drug users reporting the useof a condom the last time they had sexual intercourse[continued]17) Percentage of adults aged 15–49 who have had more than one sexual partner in the past 12 months reporting the use of a condom during their last sexual intercourse.18) Percentage of female and male sex workers reporting the use of a condom with their most recent client19) Percentage of men reporting the use of a condom the last time they had anal sex with a male partner.20) Percentage of injecting drug users reporting the use of a condom the last time they had sex.21) Percentage of injecting drug users reporting using sterile injecting equipment the last time they injected21) Percentage of injecting drug users reporting the use of sterileinjecting equipment the last time they injected
50CORE UNGASS INDICATORS: NATIONAL INDICATORS Impact22) Percentage of young women and men aged 15–24who are HIV infected*23) Percentage of most-at-risk populations who are HIV infected24) Percentage of adults and children with HIV known to be on treatment12 months after initiation of antiretroviral therapyThere are 4 indicators measuring the impact of HIV/AIDS, and provides an indication of changes in prevalence and incidence when compared to data collected in previous rounds of reporting or future rounds of reporting. The impact indicators are:22) Percentage of young women and men aged 15–24 who are HIV infected23) Percentage of most-at-risk populations who are HIV infected24) Percentage of adults and children with HIV still alive and known to be on treatment 12 months after initiation of antiretroviral therapy25) Percentage of infants born to HIV infected mothers who are infected25) Percentage of infants born to HIV-infected mothers who are infected*Millennium Development Goals indicator
51DATA PROCESS FOR UNGASS REPORTING What steps to include in the data processWhy is the data process important
52DATA PROCESS FOR UNGASS REPORTING Data collection, vetting, and analysis process including:Identification of relevant tools / sources for data collection for each indicatorTimeline for data collection in line with other data collection efforts, including those via funding agencies (e.g. GFATM, PEPFAR)Reporting timeline for facility-based data for national level aggregationData vetting workshops to reach consensus on the correct value for each indicator
53DATA PROCESS FOR UNGASS REPORTING (continued) Protocols for data processing and management including:Basic data cleaning and validationOne database for analysis and reporting purposesAnalysis of relevant datain coordination with partners from government, civil societyand the international community- See Appendix 8 Sample Checklist -
54DATA PROCESS FOR UNGASS REPORTING: the WHAT 5. Useof Data4. Consensuson Data3. UnderstandingDataHave data and related information been collected in a timely manner and collated from multiple sources for each indicator?2. GatheringHas a data process for UNGASS reportingwith roles and responsibilities been establishedand shared with all relevant partners?1. PlanningHave the characteristics and qualityof the available data been assessed and strengths and limitations understood?Has a collaborative consultation with relevant partners to reach consensus on indicator values taken place and been documented?Have a comprehensive data analysis and findings been included in the Country Progress Report, endorsed by all relevant stakeholders?This is a step diagram which poses the questions that should be asked regarding the process of the data aspects of UNGASS reporting. Details of each step are described in a handout and should be referred to when undertaking this processThe first step is planning the process: This also includes identifying indicators and if data is available, strategies for data collection, identify sources, setting timelines, in particular timelines that are harmonized with donor and other organizational reporting and data collection timelines.Second step is gathering data: Have data been collected from all existing sources, is there a point of contact for each of these sources for referral and clarification, is system set up to manage this data and related information (such as source, collection period, etc.)Third step is understanding the data: This includes knowing if the data are most recent and representative, what was the methodology for collection (was it robust), what quality control assurance was taken (does data system ensure quality, was it standardized survey, was data managed in standardized way)?The fourth step is consensus on data: Based on the results of 3. a collaborative process with relevant partners for choosing best indicator value should take place.The fifth step is use of data: Based on a comprehensive analysis of the final indicator values, interpretation and write up should be endorsed by all relevant partners. It is encouraged that this also be a collaborative process.
55DATA PROCESS FOR UNGASS REPORTING: the WHYCompleteness, integrity, credibility, and consensus of indicator data is paramount for successful monitoring of the HIV epidemic & for guiding program planning, advocacy, and policyA clear and comprehensive data process, involving all relevant stakeholders, will help ensure steps are taken to obtain the best possible data
56DATA CONSIDERATIONS Data sources Numerators and Denominators Data disaggregationRecency and representativeness of data
57DATA SOURCES Population-based surveys (e.g. DHS, AIS, MICS) Special surveys (e.g. BSS, 2nd generation surveillance surveys, school surveys, health facility surveys)Programme monitoring (e.g. patient tracking, Health Information System)Key informantsDesk reviewHIV surveillance / estimates57
58NUMERATORS AND DENOMINATORS Most indicators have a numerator and denominator to calculate a percentageFollow the instructions exactlyIt is important that the data collection period is consistent for all the information relevant to a specific indicator’s numerator and denominator58
59DATA DISAGGREGATION Countries are expected to submit total valuesANDall disaggregated values as per the instructions in the GuidelinesData collected through facilities, programme monitoring, and surveys is often disaggregated; the challenge is to ensure the disaggregated data is also available at national levelCountries are expected to submit total values AND all disaggregated values as per the instructions in the GuidelinesAs mentioned previously, less than 20% of the data submitted for the 2005 report were disaggregated, which makes it difficult to draw conclusions. It is vital that countries collect data in its component parts and not simply in its aggregated format, and to retain the disaggregated data as it moves from the local level to the national and global levels.59
60DATA DISAGGREGATION (continued) Why important?Allows for more effective tracking of resources and programmatic responsesAllows for monitoring trends in specified sub-populationsAllows for monitoring equity of access to services60
61IN-COUNTRY REPORTING PROCESS Country Progress Report FormatSubmission ProcessRole of UNAIDS/Geneva
62RECENCY & REPRESENTATIVENESS Which data to submit?Ideally,most recent data availablenationally representative dataBut,if data is not representative, most recent survey data which has been reviewed and endorsed by technical experts within the countryThe most recently available nationally representative survey should be used.It is recognised that in some cases this may mean that the data reported in this round will be the same as the data reported in 2005, since such surveys are generally undertaken at three to five year intervals.Ensuring the representativeness of samples taken for surveys of most-at-risk populations is a technical challenge. Methods are being developed to try to achieve representative sampling of these populations (i.e. respondent driven sampling). While these are being refined, it is recognised that countries may not be able to attest to the representativeness of samples used for surveys of most-at-risk populations. As such, countries are advised to report data for these indicators using the most recent survey of most-at-risk populations which has been reviewed and endorsed by technical experts within the country, such as by the M&E technical working groups or national research councils62
63STEPS FOR IN-COUNTRY UNGASS REPORTING PROCESS PreparationMonitor against action planSubmissionPlanningThis slide shows an example of the process for UNGASS reporting including preparation, planning, implementation and submission.Each of these steps will be discussed in more detail.Implementation63
64ROLES & RESPONSIBILITIES The overall responsibility of preparing and submitting the Country Progress Report lies with national governments (NAC or equivalent)UNGASS should be routine and integrated in the country’s M&E plans, budgets and systems
65PREPARATION 1. Assess country readiness 2. Identify stakeholders 3. Identify funding4. Identify relevant data tools / sources65
66ASSESS COUNTRY READINESS PreparationCountries are expectedto know their epidemicto determine which of the UNGASS indicators are applicable to the country’s epidemiological contextto know availability of data1 Countries are expected to "know their epidemic",2 To review all of the indicators in the light of this knowledge to determine which ones are applicable in the country context3 For each indicator that countries do not submit data for, countries are asked to indicate if (1) data are not available to answer that indicator, or (2) the indicator is not consider to be applicable to the epidemic situation in the country.66
67IDENTIFY STAKEHOLDERS PreparationIndividuals and organizations who are affected byHIV and/or are involved in the HIV response:Government (including NAC, Ministry of Health, Ministry of Education; Ministry of Finance)Civil society groups (including nongovernmental organizations, people living with HIV)Private sector67
68IMPORTANCE OF CIVIL SOCIETY PreparationStrategic and tactical expertiseQuantitative and qualitative dataValuable perspective on the dataCivil society plays a key role in the response to the AIDS epidemic in countries around the world. The wide range of strategic and tactical expertise within civil society organizations makes them ideal partners in the process of preparing National Progress Reports. Specifically, civil society organizations are well positioned to provide quantitative and qualitative information to augment the data collected by governments. They can provide a valuable perspective on the issues included in the National Composite Policy Index. They are also equally well positioned to participate in the review and vetting process for progress reports.
69HOW AND WHEN TO INCLUDE CIVIL SOCIETY PreparationInvolvement in all phases of the report preparation:Identify relevant civil society organizationsAssign roles and responsibilityPlan with them data collection and analysisInvite them to participate in workshopsShare drafts with them for review and commentsDisseminate final report to them69
70IDENTIFY SOURCE OF FUNDING PreparationWhat sources are available?No separate funding available for UNGASS reporting, but should be integrated in M&E budgets (government, GFTAM, World bank)
71DATA SOURCES Population-based surveys (e.g. DHS, AIS, MICS) PreparationPopulation-based surveys (e.g. DHS, AIS, MICS)Special surveys (e.g. BSS, 2nd generation surveillance surveys, school surveys, health facility surveys)Programme monitoring (e.g. patient tracking, Health Information System)Key informantsDesk reviewHIV surveillance / estimates71
72DATA CONSIDERATIONS Recency and representativeness of data PreparationDATA CONSIDERATIONSRecency and representativeness of dataRequirements for disaggregationAppropriate data process72
73PLANNINGPlanningDevelop a plan for data collection / collation, analysis and report writingDesignate a coordinatorAssign roles and responsibilities for each indicatorAssign realistic timelines and milestonesSecure funding for the entire process (collecting, collating, analysing and reporting the data)73
74PLANNING (continued) Disseminate the plan Use the plan as a checklist to monitor progress against the timeline and milestones74
75ACTION PLAN TEMPLATE Steps Activities Timeline Budget PlanningACTION PLAN TEMPLATEStepsActivitiesTimelineBudgetResponsible personStakeholders involved75
76IMPLEMENTATIONImplementationEstablish a process for data collecting / collating, vetting and analysisOrganise workshops for vetting and triangulation of dataComplete the appropriate data formsDraft the Country Progress ReportDisseminate it for review and commentCollate comments from stakeholders, including government agencies and civil society76
77IMPLEMENTATION (continued) Organize a national consultation workshop to reach consensus with stakeholders on the final reportEnter agreed data into CRIS (or equivalent data management system)Validate the indicator data against the report narrative (consistency check)Finalize and submit the report to UNAIDSIdentify focal point for communication between UNAIDS Secretariat and country77
78MONITOR PROGRESS & DEBRIEF Monitor progress throughout the implementation process using the action planPlan a debriefing meetingto assess success of report (timeliness, comprehensiveness, quality)to agree actions to undertake to strengthen the national M&E system78
79OVERVIEW OF THE REPORTING PROCESS Identify data needs, data sources, stakeholders, fundsEstablish plan for data collection, analysis and reporting writingSecure funds, collect/collate and analyse data, complete data formsDraft Country Progress Report, share draft with stakeholdersEnter data in CRISThis figure shows an overview of the steps of the reporting process we have just gone through. Starting with identification of data, data sources, stakeholders and funding; before establishing an action plan for the process. Then the data collection, collating and analysis should be carried out after the funding is in place. The country progress report should be drafted using the data collected, and then shared with stakeholders for input and consensus. The next step is to enter the data into CRIS and the narrative report should be validate against the CRIS data. Before submission there should be consensus with the stakeholders about the data. It is important to stress that once the data have been submitted, they can not be changed so, it is important to make sure that the data are accurate. The last step is submission of the data and the narrative part of the Country Progress Report.Validate the narrative report against the data in CRISReach consensus with stakeholders on final Report & submit timely
80FORMAT OF THE COUNTRY PROGRESS REPORT Submission of reportNarrative part (see Appendix 2)Structured to ensure consistent information across countriesProvide an in-depth analysis of data regarding the HIV epidemic and responseThere are two parts of the Country Progress Reports that should be submitted to UNAIDS, namely a narrative part (including the NCPI as an annex) and the UNGASS indicator data. The narrative report should follow the suggested format that is provided in Appendix 2 in the Guidelines. The template for the country progress report has been made more structured for this round of reporting, to ensure consistency of information across countries.80
81UNGASS COUNTRY PROGRESS REPORT [Country Name] Reporting period: January 2006–December 2007Submission date: […]Table of ContentsStatus at a glanceOverview of the AIDS epidemicNational response to the AIDS epidemicV. Best practicesVI. Major challenges and remedial actionsVII. Support from country’s development partnersVIII. Monitoring and evaluation environmentThe template for the UNGASS country progress can be found in the guidelines as appendix 2. The sections of the report are:II. Status at a glace: this section should provide a brief summary of(a) the inclusiveness of the stakeholders in the report writing process;(b) the status of the epidemic;(c) the policy and programmatic response; and(d) UNGASS indicator data in an overview tableIII. This section should cover the detailed status of the HIV prevalence in the country during the period January 2006–December 2007 based on sentinel surveillance and specific studies (if any) for the UNGASS impact indicators. The source of information for all data provided should be included.IV. This section should reflect the change made in national commitment and programme implementation broken down by prevention, care/treatment and support, knowledge and behaviour change, and impact alleviation during the period January 2006–December 2007.Countries should be specifically address the linkages between the existing policy environment, implementation of HIV programmes, verifiable behaviour change and HIV prevalence as supported by the UNGASS indicator data. Where relevant, these data should also be presented and analysed by sex and age groups. Countries should also use the National Composite Policy Index (see Appendix 7). Countries are encouraged to report on additional data to support their analysis and interpretation of the UNGASS data.V. This section should cover detailed examples of what is considered a best practice in-country in one or more of the key areas (such as political leadership; a supportive policy environment; scale-up of effective prevention programmes; scale-up of care, treatment and/or support programmes; monitoring and evaluation, capacity-building; infrastructure development. The purpose of this section is to share lessons learned with other countries.]81
82ANNEXES ANNEX 1 Consultation/preparation process for the Country Submission of reportANNEXESANNEX 1Consultation/preparation process for the CountryProgress ReportANNEX 2National Composite Policy Index QuestionnaireThere are two annexes to the country progress report and they should both be included in the narrative report when submitted:Annex 1 is the consultation/preparation process for the Country Progress ReportAnnex 2 is the National Composite Policy Index Questionnaire
83COUNTRY PROGRESS REPORT Indicator Data CRIS is the Country Response Inforamtion System
84BENEFITS OF USING THE COUNTRY RESPONSE INFORMATION SYSTEM (CRIS) Submission of reportBENEFITS OF USING THE COUNTRY RESPONSE INFORMATION SYSTEM (CRIS)CRIS is a monitoring system which greatlyfacilitates UNGASS reporting:It supports all six UN languages [no need for translation]It uses the current Guidelines [no confusion]It uses the correct indicator definitions [no confusion]It facilitates compilation and management of all UNGASS data [saves time]
85BENEFITS OF USING CRIS (continued) Submission of reportBENEFITS OF USING CRIS (continued)It has built-in validity checks [avoids logical errors]It stores available data from population-based surveys [saves time]It facilitates trend analysis as previous UNGASS data is stored [saves time]Data is seamlessly integrated into the global database [no transcription errors, no delays]
86BENEFITS OF USING CRIS (continued) Submission of reportBENEFITS OF USING CRIS (continued)In addition to use for UNGASS reporting andreporting to donorsCRIS can be used for the management of allmonitoring information at the nationaland sub-national levels
87SUBMISSION PROCESS What? Country Progress Report consisting of When? Submission of reportWhat?Country Progress Report consisting ofNarrative section (including Annex 1, Annex 2)CRIS data file (or excel data file, see template on CD)When?15 January 2008 if not using CRIS31 January 2008 if using CRIS [preferred]How?to
88WHAT HAPPENS AT UNAIDS/GENEVA? UNAIDS and its partners are responsible for calculating the global indicatorsA process for collating, analysing and reporting data in coordination with co-sponsors and global partners, including civil society occurs88
89WHAT HAPPENS AT UNAIDS/GENEVA? Country Report Findings + Global indicators=Report on the Global AIDS EpidemicThe country data together with the global indicators will then be used to produce the report on the Global AIDS epidemic.89
90WHAT HAPPENS AT UNAIDS/GENEVA? The Report on the Global AIDS Epidemic is prepared by May 2008 and submitted to the United Nations General AssemblyThe Country Progress Reports and the Global Report will be available on the UNAIDS website at90
91BENEFITS OF UNGASS REPORTING Provides answers to important questions:What is the status of the epidemic in the country?What are the basic trends in HIV transmission and service coverage?What are the main obstacles to accessing HIV prevention, care and treatment services?What is the quality of services being delivered?91
92BENEFITS OF UNGASS REPORTING (continued) Provides answers to important questions:Are services being delivered equitably and effectively?What exacerbates these problems? (e.g. policies, laws, resources, politics, customs, organizations, individuals)Who can change this situation? (e.g. elected leaders, bureaucrats, religious leaders, community leaders, traditional leaders, donors, international organizations, NGOs)What are these people/organizations currently doing to address the problems?92
93BENEFITS OF UNGASS REPORTING (continued) This information can be used toevaluate the effectiveness of the national responseanalyse linkages between policy, implementation of programmes, behavioural change and HIV prevalenceidentify gaps in programmes, personnel, technologyidentify data gaps and data quality issuesstrengthen the National M&E system93
94SUMMARY Ownership & participation Consultation & consensus Data vetting, validation, triangulationTimeliness: 31 January 2008or 15 January 2008 if not submitted in CRISFocus on systems and processTo summarize what is important to the UNGASS reporting process:- Ownership & participation throughout the whole process by both government and civil society- Consultation & consensus during the entire process- Data need to be vetted, validated, and triangulated before submission. After submission, the data can not be changed.- Timeliness: 31 January 2008(or 15 January 2008 if not submitted in CRIS)- Focus on systems and process94