Monitoring and Evaluation: A Review of Terms Lisa V. Adams, MD E&E Regional Workshop Kiev, Ukraine May 23-26, 2006

Goals To provide better treatment for people with tuberculosis in Country X To achieve a treatment success rate of 85% in Country X by the end of 2006 The core of any M&E system is the goals and objectives of the program. If they are written in such a way that they can be easily distinguished from one another and measured, the job of the m&e specialist will be much easier. Goal: statement that describes the hoped for result of the program. Goals are achieved over the long term and through the combined efforts of multiple programs. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

Objectives To increase the number of pharmacies with adequate supplies of drugs To increase to 100% the proportion of pharmacies implementing NTP guidelines on drug management by the end of 2005 Specific and operationalized statements detailing the desired accomplishments of a program (Rossi & Freeman, 1993) A properly stated objective is action oriented and starts with the word: to and is followed by an action verb. It addresses what and when, but not how or why. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

Objectives – Be SMART Specific Measurable Achievable Realistic Time-Bound note to trainer: useful to go through examples of each one of these components Specific – does it cover only one rather than multiple activities? Measurable – can it be counted in some way? Achievable– can we achieve this with available resources? Is it doable? Realistic – we would all love to detect and cure every case of TB in the world. Time-Bound– does it give a time frame? Transition – now we will discuss specific program components and how they described in monitoring and evaluation terms. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

What do you need to implement the program? Inputs What do you need to implement the program? A collection of resources (financial, personnel, equipment) which are the raw materials for the program Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

What do you need to implement the program? Inputs What do you need to implement the program? Drugs and supplies Clinical sites Laboratories Nurses, doctors, epidemiologists, etc DOTS strategy, policies supporting DOTS Money A collection of resources (financial, personnel, equipment) which are the raw materials for the program Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

WHAT DOES THE PROGRAM DO? Process WHAT DOES THE PROGRAM DO? A set of activities in which program inputs are utilized in pursuit of the results expected from the program. Notice, these can be concrete, tangible things or can be conceptual– supervision, for example. Usually action oriented, for example, training or supervision activities that will produce something. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

WHAT DOES THE PROGRAM DO? Process WHAT DOES THE PROGRAM DO? Diagnosis and Treatment Meetings Training Workshops Reporting & Recording Supervision A set of activities in which program inputs are utilized in pursuit of the results expected from the program. Notice, these can be concrete, tangible things or can be conceptual– supervision, for example. Usually action oriented, for example, training or supervision activities that will produce something. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

WHAT DID THE PROGRAM ACHIEVE? OUTPUTS WHAT DID THE PROGRAM ACHIEVE? The results obtained at the program level through the execution of activities using program resources. When we think of outputs, we ask – what did the program achieve? Did the inputs and processes produce tangible items or outputs that can be quantified. Ask the group for examples of outputs…then show these examples. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

WHAT DID THE PROGRAM ACHIEVE? OUTPUTS WHAT DID THE PROGRAM ACHIEVE? Volume of drugs distributed to regional storage facilities Number of primary health care centers implementing directly observed therapy Number of laboratory technicians trained in smear microscopy Number of TB cases tested for HIV The results obtained at the program level through the execution of activities using program resources. When we think of outputs, we ask – what did the program achieve? Did the inputs and processes produce tangible items or outputs that can be quantified. Ask the group for examples of outputs…then show these examples. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

Outcomes WHAT ARE THE GOALS OF THE PROGRAM? Case Detection Smear Conversion Treatment Outcomes Effects of a program at the target population level (for example, cohort analysis) These outcomes are standardized for global reporting to WHO. They tell us the results of our efforts for the cohort of patients detected and treated by the NTP. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

Medium and Long Term OUTCOMES Outcomes for overall DOTS implementation Case detection, Smear conversion, Treatment success Medium term outcomes for NTP implementation - Proportion of TB cases administered directly observed therapy However, sometimes it is useful to think of outcomes in other ways. Long vs. medium term outcomes – sometimes it is helpful to think of long term outcomes or the overall outcomes of DOTS implementation separately from outcomes we hope to achieve through all of the NTP’s activities, which may be observed in the short term and directly related to program outputs. For example, long term outcomes may be the program results we hope to achieve with regards to case detection, smear conversion, and treatment outcomes, while medium term outcomes may be directly related to the implementation of one component of the DOTS strategy, for example, administration of directly observed therapy. We’ll give an example of this towards the end of the presentation to link this all together. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

WHAT IS THE EFFECT OF THE PROGRAM ON THE EPIDEMIOLOGY OF TB? IMPACT WHAT IS THE EFFECT OF THE PROGRAM ON THE EPIDEMIOLOGY OF TB? Transmission dynamics of TB Incidence of pulmonary TB Changes in mortality due to TB Decrease in deaths due to HIV-related TB Prevention of MDR-TB Looking at the impact of our program answers this question: what is the effect of the program on the epidemiology of TB? AND most important, are you able to link these directly to your program efforts? Show examples of impact – note that the epidemiology of TB is affected by many factors, not just our program, and this is one way we can distinguish outcome from impact. All of these items may be affected by other health programs or interventions or phenomena that are not addressed by TB programs. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

Monitoring vs. Evaluation Inputs, Process, Outputs, Outcomes: Monitoring Impact: Evaluation Here’s another way of thinking about it. These terms are related to the difference between monitoring and evaluation, covered in the first presentation. Inputs, processes, outputs, outcomes are measured with monitoring activities and usually with data from your program and patients. Impact is measured with evaluation activities and involves data sources beyond the routinely reported program and patient data Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006

KEY TERMS – Directly Observed Therapy INPUT – NTP guidelines for directly observed therapy (DOT) PROCESS – NTP training workshops on how to implement DOT OUTPUT – Number of nurses trained on administration of DOT Intermediate OUTCOME – Proportion of smear positive TB cases on DOT increases Long term OUTCOME – Treatment success rate increases to 85% IMPACT – Decrease in morbidity and mortality due to TB and prevention of MDR -TB Here is an example of how to apply these terms to NTP activities. NTP manager wants to monitor and evaluate the implementation of directly observed therapy to smear positive TB cases. What does he or she NEED to accomplish this goal? (CLICK) Inputs – a supportive policy/guidelines developed and endorsed by the NTP and key partners. What types of activities will he or she need to DO to accomplish this goal? (CLICK) Process – Training workshops for doctors and nurses on how to implement DOT in the clinic What will this ACHIEVE? (CLICK) Outputs – The NTP will report on the number of nurses trained in administration of DOT. How are these inputs, processes, and outputs linked to NTP GOALS? (CLICK) Medium term OUTCOME – the proportion of smear positive TB cases receiving DOT will increase Long term OUTCOME – the treatment success rate will increase to 85% Finally, what is the effect of this NTP activity on the population? (CLICK) IMPACT – Morbidity and mortality due to TB will decrease and we will prevent the emergence of MDR-TB. Tuberculosis Technical Focus Area Activties and Update All-Staff Meeting 2006