Monitoring and Evaluation: Tuberculosis Control Programs
Published byModified over 6 years ago
Presentation on theme: "Monitoring and Evaluation: Tuberculosis Control Programs"— Presentation transcript:
1 Monitoring and Evaluation: Tuberculosis Control Programs
2 Learning ObjectivesUnderstand the principles of M&E for effective TB programming.Construct conceptual and result frameworks.Select and make proper use of indicators and data for TB M&E.Be able to develop a monitoring and evaluation plan.
3 Content Outline Problem statement M&E (definitions) Opportunities, challenges, and strategies for TB controlConceptual and results frameworksM & E frameworks
4 Content Outline…cont’d TargetsM & E indicatorsSource of dataM&E tools for TLCPM & E challenges
5 Problem statement1/3 of world population (2 billion) infected with M. tuberculosis.9 million new cases of TB/year.2 million deaths/year.Inadequate Control Programmes.Speaker noteAccording to WHO, TB infection is currently spreading at the rate of one person per second. It kills more young people and adults than any other infectious disease and is the world’s biggest killer of women. Every year 8-10 million people catch the disease and 2 million die from it. About a third of the world’s population, or around 2 billion people, carry the TB bacteria but most never develop the active disease. Around 10% of people infected with TB actually develop the disease in their lifetimes, but this proportion is changing as HIV severely weakens the human immune system and makes people much more vulnerable.
7 M & EMonitoringis the routine tracking of programs using input, process and outcome data that are collected on a regular basis.is used to assess whether or not planned activities are carried out according to schedule.is usually done by insiders.
8 M&E …cont’d Evaluation Periodic assessment of programme or project against set targets.Usually done by outsiders.TypesProcess evaluationOutcome and impact evaluation
9 M&E…cont’d Process evaluation is used to measure quality and integrity of programme implementation and to assess coverageit may also measure the extent to which the intended target population uses servicesinform midcourse corrections in the programme
10 M&E…cont’d Outcome evaluation measures the extent to which stated objectives are achieved with respect to the programme’s goalsassesses influence of programme activities by measuring changes in knowledge, attitude, behaviors, skills, community norms, and health-service utilization.
11 M&E…cont’d Impact evaluation is used to determine how much the observed change in outcomes can be attributed to specific programme efforts.involves complex data collection and analysis proceduresassist to determine the success of a project for scale-up or replication.
12 Why M &E? M& E assists in day-to-day management of health programmes. M&E provides information for strategic planning, programme design and implementation.M&E assists informed decision-making about human and financial resources, especially in resource-limited settings.
13 Good M&Eensures the most efficient use of resources to generate the data needed for decision-making.guides data collection and analysis to increase consistency and to enable managers to track trends over time.serves as a catalyst to coordination.
14 Opportunities for TB Control Low cost, accurate diagnosis and treatment available for over three decades.M & E system is in place.Speaker’s noteUnfortunately, despite the availability of low-cost and accurate diagnosis as well as nearly 100% curative treatment for more than three decades, tuberculosis remains one of the leading infectious causes of death globally, killing nearly two million people a year.It is sad to note that there was no rapid progress in the field of tuberculosis control over the past two decades. During the last three decades no single anti-TB was discovered. As a result it was only the level of frustration that grew. However, in recent years, there has been renewed interest in tuberculosis.
15 Challenges of TB Control Global emergency- Rising incidence of TB.- HIV pandemic.- MDR- TB.Gaps in coverage, case detection and treatment successSpeaker notesThe World Health Organization (WHO) declared tuberculosis (TB) a global emergency in 1993, in response to a steady increase in the incidence of TB, shifting dynamics in TB disease related to the HIV/AIDS epidemic, and the emergence of multidrug- resistant TB (MDR-TB). The increasing burden of TB is due to many factors, including neglect of TB control by governments; poor management of programmes;In response to this emergency, the WHO developed a strategy known as DOTS (the internationally recommended TB-control strategy). The DOTS strategy is designed to correct weakness in previous models of program management and to strengthen diagnosis and treatment services.TB control programmes face many new and existing challenges. Traditionally, a lack of political commitment to TB control, which in turn leads to weak support of TB control activities from the health system and society, continues to be an ongoing challenge in many countries. Similarly, weak public sector health services, which desperately need to enhance their capacity to implement, expand, and sustain DOTS- based services without compromising the quality of case detection and treatment, hinder progress in TB control.Among the newer challenges, the impact of the HIV/AIDS epidemic on TB incidence is daunting. Even in the presence of well-functioning TB control programmes, the incidence of active disease is increasing in settings with a high prevalence of HIV.Another challenge is the exponential increase in MDR-TB. This challenge requires effective implementation of the DOTS strategy to prevent new MDR-TB cases.Although some progress has been made, persistent gaps remain in coverage, case detection, and treatment success – three key global indicators that are recommended by the World Health Assembly for measuring national TB control program (NTP) success.As a result of all these challenges TB remains to be a leading infectious cause of death globally.
16 Control Strategy (DOTS) Sustained political commitment.Access to quality-assured TB sputum microscopy.Standardized short-course chemotherapy.Uninterrupted supply of quality-assured drugs.Recording and reporting system enabling outcome assessment.Speaker’s noteDOTS is the internationally-recommended strategy to ensure cure of tuberculosis. It is based on five key principles that are common to disease-control strategies, relying on early diagnosis and cure of infectious cases to stop spread of tuberculosis.The package of interventions that eventually became to be known as the DOTS strategy was first formulated in national tuberculosis programmes supported by the IUATLD under the leadership of Dr Karel Styblo. Initially in the United Republic of Tanzania and then later in several other countries of Africa and Latin America.The system of recording and reporting under the DOTS strategy has three essential components:- recording all patients with the diagnosis of TB.- recording their treatment outcome.- reporting results to the next higher level.WHO began to promote this strategy in 1991 and in 1994 produced a framework later known as the DOTS strategy.
17 Basic Assumptions for DOTS Government commitment avails sufficient funds and administrative support.Microscopic exams detect the most infectious cases and are affordable.Direct observation ensures adherence.Uninterrupted drugs ensure cure.Recording & Reporting help to monitor and evaluate.Speaker’s noteGovernment commitment is an essential component of DOTS, and WHO has emphasized advocacy and social mobilization as means of achieving this commitment. Sufficient funds and administrative support to hire staff, purchase essential items (drugs, microscopes, reagents, printed materials, etc), and contract for services are necessary for the programme to operate.The rationale for diagnosis primarily by microscopy among patients in health facilities is simplicity of the procedure (i.e., it can be done in remote areas with minimum lab facilities and detects the most infectious cases).Direct observation of treatment in which “ a trained and supervised person observes the patient swallowing the tablets” is fundamental to the DOTS strategy to ensure adherence to treatment.The requirement of an interrupted supply of tuberculosis drugs is clear. In addition, the quality of drugs should be ensured, particularly if they are provided in fixed dose combinations, which are more susceptible to problems in manufacture.The last component is the main source of epidemiological data useful to monitor and evaluate the control programme. DOTS records can be easily checked for internal consistency and for consistency between records, and can also be externally verified by reviewing sputum slides, interviewing patients and health workers, and monitoring consumption of drugs and supplies.
18 Levels of intervention for TB Control Primary – BCG vaccination- INH prophylaxisSecondary – early diagnosis and proper treatmentTertiary – Prevent complicationsSpeaker’s noteBCG vaccination protects against severe forms of Tuberculosis (TB meningitis and miliary TB) and provides more protection against Leprosy than TB.INH prophylaxis is administered to persons not yet infected, to prevent infection, because if that should occur, the person’s natural defenses would put little resistance. This is especially indicated for children in contact with a smear positive patient, up to two months after that patient has become bacteriologically negative as a result of treatment (primary chemoprophylaxis).INH is also indicated in persons already infected, to reinforce natural defenses especially for persons with immunosuppression (secondary chemoprophylaxis). INH was selected due to its low cost, few adverse effects, oral administration and efficacy.Some cases of TB (TB of the vertebrae benefit from surgical interventions to prevent neurological sequale).TB prevention and control relies primarily on secondary level of intervention. This implies early diagnosis and proper treatment of cases. At individual level it helps to treat and cure the patient before his/her condition gets worse and at community level it helps to prevent spread of infection to the community.
19 Conceptual Frameworks – TB Programmes External Factors ResourcesClinical and managerial staffDrugsLaboratoriesTB infectionHealth Systems (DOT)AvailabilityAccessQualityUtilizationTB MorbidityPrevalenceIncidenceHIV co-infectionMDR-TBTB mortalitySpeaker NotesThis is an example of conceptual framework for national tuberculosis programmes.Factors include those external to a programme, such as demographic or socio-economic characteristics. It also includes factors related to the health system and to individuals.You can see that the programme is aiming to affect the health system and knowledge of TB which will in turn affect morbidity, which will ultimately affect mortality due to TB. Other health related factors that can affect morbidity are included in this model.Program FactorsPolitical commitmentDonor involvementNational TB programmeTB knowledgeCase detectionAdherenceStigmaCo-morbidityHIV MalnutritionAlcoholism Diabetes
20 M&E framework for TB programme OUTPUTDiagnostic & Treatment servicesImproved KAPReduced StigmaOUTCOMECase detectionCase treatmentCase holdingIMPACTTB infectionTB morbidityTB mortalityINPUTPolicy environmentHuman and financial resourcesInfrastructurePROCESSNTP MgtTrainingDrug MgtLaboratoriesACSAn M&E framework is a visual conceptualization of how the elements of a programme fit together, that is , which inputs are necessary for the programme’s activities (process), what outputs are expected from the activities, and what short-and long term outcomes will ultimately result from the programme.A framework can be used as a tool to understand and analyze a programme, which is crucial for developing and implenting sound M&E plans.Here is a generic example of an M&E framework for a national TB programme. This framework shows the linear relationships between inputs, processes, outputs, outcomes, and impact. It also provides a visual interpretation of the broad context for the NTP, which includes: Political commitmentHealth system variablesSocio-economic conditionsEpidemiological context of TBThe purpose of the framework is to expand on it and create a M&E framework that could be used to guide planning, monitoring and evaluating the directly observed therapy as the standard of care for smear positive TB cases.CONTEXTPolitical commitment Health system Socio-economic conditions Epi-contextAvailability HIV prevalenceAccess MalnutritionUtilization Alcoholism
21 Results Frameworks - TB programmes SO1: Increase tuberculosis case detection to 70%IRl: Increased availability of quality servicesIR2: Increased demand for quality servicesIRl.1: Services increasedIR2.1: Customer knowledge of TB improvedSpeaker NotesSuccessful TB control calls for increased case detection, and that in turn calls for increasing demand and supply.Note that in this more developed example, the IRs and sub-IRs are phrased in such a way to indicate the desired results. For example, IR1 is not simply “quality services” , but “availability of quality services increased.”Also note that each IR and sub-IR is measurable. That is, indicators can be developed and data collected to calculate those indicators.IR : Intermediate ResultIRl.2: Practitioners’ skills and knowledge increasedIR2.2: Social support for TB practices increasedIRl.3: Improved programme management
22 Global Targets (by 2005) Case detection Detect 70 % of all smear positive TB casesTreatment outcome85% of detected TB cases are curedSpeaker’s noteThe global targets for tuberculosis control are to detect 70% of the estimated new sputum smear-positive cases and to cure 85% of the sputum smear-positive cases detected. These targets were originally adopted by WHO in It became clear that the global targets would not be achieved by 2000 as intended, and the target date was postponed to 2005 by the World Health Assembly in May 2000.The numerator for the case detection target is the number of new cases of sputum smear-positive tuberculosis registered in one year, and the denominator is the number of new sputum tuberculosis cases estimated to have arisen in the same population over the same period. As the case detection target relies on an estimate for the incidence of tuberculosis, it is difficult to measure in most settings, especially in the context of an epidemic of HIV infection.The numerator for the cure-rate target is the number of patients in one-year cohort of new cases of smear-positive tuberculosis fulfilling the WHO definition of cure, and the denominator is the number of patients originally registered for treatment in that cohort.Adoption of the targets is based on two principles – impact and feasibility. First, epidemiological modeling has demonstrated that achieving the targets will result in a significant decline in the tuberculosis epidemic, reducing incidence by about 50% in 8-12 years, in the absence of HIV. Second, the targets took into account accessibility and affordability of services and health-seeking behavior of the population.
23 Indicators Valid Reliable Specific Sensitive Operational Affordable FeasibleComparableValid: indicators should measure the condition or event they are intended to measure.Reliable: indicators should produce the same results when used more than once to measure the same condition or event, all things being equal (e.g., using the same methods, tools , or instruments).Specific: indicators should measure only the condition or event they are intended to measure.Sensitive: indicators should reflect changes in the state of the condition or event under observation.Operational: indicators should be measured with definitions that are developed and tested at the program level and with reference standards.Affordable: the costs of measuring the indicators should be reasonable.Feasible: it should be possible to carry out the proposed data collection.Comparable: indicators should be comparable (e.g., over time, across geographical lines).
24 MDGs (by 2015)Goal 6 : to combat HIV/AIDS, malaria, and other diseasesTarget 8: to have halted and begun to reverse the incidence of malaria, TB, and other major diseases by 2015Indicator 23: between 1990 and 2015, to halve the prevalence and death rates associated with tuberculosis; andIndicator 24: by 2005, to detect 70% of smear positive and successfully treat 85% of these cases.Speaker’s notesThe MDG framework consist of a hierarchy of indicators that measure progress towards “targets”, which are the specific achievements needed to satisfy higher “goals.” Those most directly relevant to TB control are Goal 6 (to combat HIV/AIDS, malaria and other diseases) and Target 8 (to have halted by 2015 and begun to reverse the incidence of malaria, TB, and other major diseases). Among the indicators for Target 8 are two groups that can be used to evaluate the implementation and impact of TB control: indicator 23 and 24.The target for case detection and treatment have been set by WHO’s World Health Assembly (Resolution WHA 44.8).The targets for prevalence and deaths are based on a resolution of the year 2000 meeting of the Group of Eight (G8) industrialized countries, held in Okinawa, Japan.
25 Global Indicators TB case detection. Treatment success rate. DOTS coverage.Surveillance of multi-drug resistant TB.HIV seroprevalence among TB patients.Speaker’s noteThe five global indicators described in this section are based on data reported by National TB Programmes. Data are used to monitor progress in DOTS expansion and achievement at national and global levels of the WHO targets for TB control: case detection rate of at least 70% and treatment success of at least 85%.The first three indicators are reported to WHO every year by national TB programmes and are included in the annual WHO report on global TB control. These indicators measure NTP progress towards international targets for case detection, treatment success, and DOTS coverage.The last two indicators provide important information on whether countries are aware of the prevalence of MDR-TB and HIV among TB cases. It is important to monitor whether or not NTPs are performing surveillance to estimate the prevalence of HIV among TB cases, and vice versa, because these data is needed to make decisions with regard to collaborative programs.National surveillance is an important tool for monitoring trend in epidemiological indicators. These trends will in turn indicate the success of control programme. These indicators are, critical to monitoring, evaluation, and problem-solving at national and local levels.
26 Programme-outcome indicators Case-notification rate (all forms of TB)Case-notification rate (new smear-positive cases)Re-treatment of TB casesSmear-conversion rateCure rate, Treatment-completion rateTreatment-failure rateDefault rateDeath rateSpeaker’s noteThe indicators listed above are routinely calculated by TB control programmes at district, regional , and national levels, and they are based on data from the TB registers and quarterly reports on TB case registration, smear conversion, and treatment outcomes. They are used to monitor progress towards achievement of the national targets for case detection and treatment outcomes and to monitor programme quality and effectiveness.Indicators are used as tools to measure whether patient-oriented activities and programme management activities are reaching their target.Outcome indicators are used to monitor and evaluate performance. They are considered the optimal level of achievement. However, they should not become a fixed ‘target’, which may lead to falsified data or neglect of one function in favour of other priorities.Due to the difficulty in measuring the epidemiological impact, outcome indicators are often used as a surrogate for indicators of impact.
27 Sources of Information Record forms at the health facilityRecord and report forms at the district levelLaboratory recordsReport forms at the regional levelReport forms at the national levelSpeaker’s notesOnce a TB programme has designed and adopted an M&E framework and selected the appropriate indicators, data-collection strategies need to be selected. There is a variety of methods that are typically used to gather TB information. No single data source can provide all of the information required for M&E – a combination is necessary.Routine data collection at TB treatment facilities and microscopy units is the most common way of collecting TB data for patient and treatment facility management, for monitoring resources used and services provided, and for disease surveillance. Data are recorded by the health staff at the facility or microscopy units while they perform their daily health care activities. These data are recorded on standard reporting forms, which are sent to basic management units (BMUs), where they are aggregated and sent to the national level.The district, regional, and national TB offices are responsible for their respective geographic areas. Monitoring is often required on a monthly or quarterly basis using several different data-collection tools. The forms listed above are the most commonly-used ones.
28 M&E tools for TLCP Supervision checklist - checklist for programme management- checklist for health facilityReview meeting- annual and semi-annual- central, regional and districtExternal Quality Assurance
29 Additional sources of Information (Special studies) Prevalence surveysPopulation-based surveysHealth-facility surveysVital registration surveysTuberculin surveysDrug-resistance surveysSpeaker’s noteSpecial surveys or studies may be needed to determine many of the epidemiological and behavioral indicators that are not collected through the routine recording and reporting. Such studies are often more comprehensive than standard collection, but at the same time, they are more costly and require specific technical capacity for implementation. These factors limit the number of special studies that are conducted.
30 M & E challenges in TB Incomplete recording and reporting Inconsistent data collectionLack of timelinessInappropriate use of informationSpeaker noteAn M&E system is only as good as the data that are collected. The data should be appropriate, complete, consistent, and timely. Many current efforts at data collection, particularly those conducted routinely, result in poor-quality data because of a lack of proper training and supervision. If the individuals recording the data are not using the data and do not fully appreciate data needs for programme management beyond the facility level, the quality will most likely be poor. This in turn leads to declining use. One of the key functions of an M&E system is to oversee all data collection and ensure that data are appropriately used and the results are disseminated throughout the system, but especially to the collection level. Changes in health programs that are directly based on evidence from the field reinforce the efforts at the peripheral level to complete routine reporting. When health workers understand the importance of the data they are collecting, quality is likely to improve, building more confidence in and use of monitoring data.
31 Level of M&E in TB: The “ONION” estimated TB casesall true TB casescases presenting to health facilitiescases presenting to public health facilitiescases presenting to DOTS facilitiescases correctly diagnosed by DOTS facilitiesdiagnosed cases reported by DOTS facilitiesThis “onion”, developed by Chris Dye of the World Health Organization, is one way of illustrating how monitoring and evaluation differ, and why we need monitoring and evaluation to guide our programmes.With a good M&E system, we will be able to track programme successes for the core of the onion and the first or second layer through monitoring activities by use of existing R & R system. The outermost layers require more than monitoring data – we will need an evaluation with rigorous methodology to determine if our programme is having any effect at this level.With quality R&R the reported number of cases reflect the number diagnosed. With good quality of service (i.e., diagnostic service) the number of cases presenting to DOTS and those correctly diagnosed is similar. Availability and strength of PPM-DOTS determine the proportion of cases presenting themselves out of all presenting to all health facilities. Accessibility, affordability and health-seeking behavior determine the proportion of cases presenting themselves among the true cases.Chris Dye, 2002
32 ReferencesCompendium of Indicators For Monitoring And Evaluating National TB Programmes. Stop TB Partnership August 2004.2. Toman’s Tuberculosis Case Detection, Treatment, And Monitoring. Second Edition WHO Geneva 20043. WHO REPORT 2005 GLOBAL TB CONTROL Surveillance, Planning, Financing