MONITORING AND EVALUATION IN TB/HIV PROGRAMS Presentation for the ProTEST Lessons Learnt workshop, Durban, RSA By Wilfred Nkhoma Medical Officer, WHO/AFRO/TB UNIT

FOCI OF MONITORING AND EVALUATION IN TB/HIV PROGRAMS Assessment of the design of interventions Monitoring of program implementation Assessment of program utility (effectiveness and efficiency):

Monitoring program conceptualisation Has a problem been appropriately conceptualized ? is the program designed to meet intended objectives ?)

MONITORING PROGRAMME IMPLEMENTATION: To determine the extent to which a program is reaching the appropriate target population [Program coverage] To determine whether or not delivery of program elements (services and treatments) is consistent with program design specifications[Program delivery] To determine what resources are being expended in the conduct of the program [Program costs]

Monitoring of Program Implementation questions There is no point in being concerned with the impact or outcome of a program unless it has indeed taken place and served the appropriate participants in the way intended (as designed). [One not uncommon reason impact evaluations reveal little or no impact is not that a program is ineffective, but rather that its implementation is faulty or incomplete] Is the program reaching the specified target population (program coverage)? Are the intervention efforts being conducted as specified in the program design (program delivery)

Monitoring programme coverage Coverage refers to the extend to which the participation by the target population reaches the levels specified in the program design [conclusions include undercoverage, bias and overcoverage].Bias is the degree to which sub-groups of the target population participate differentially Estimating of efficiency of programme coverage: Coverage efficiency = 100 *{number Number not } {in need served - in need served } {Total number Total number } {in need served }

Monitoring delivery of services Measures the extent to which program specifications are met in the delivery of the interventions. Common causes of delivery failures (Rossi, 1978): No treatment or not enough is delivered: “Nonprograms”. Little evidence that a program existed ! Wrong treatments delivered Treatment is unstandardised, uncontrolled or varies across target populations [e.g too much discretion in the hands of service providers!] resulting in non-comparability between sites or communities

ASSESSMENT OF PROGRAM UTILITY Assessing the utility of intervention programmes involves systematic examination of : i) Programme impacts [extent to which programme produces desired outcomes] ii) Programme efficiency [the benefits of a programme in relation to its costs] Thus utility assessments help to distinguish useful from ineffective and inefficient intervention programs/ interventions

IMPACT ASSESSMENTS Undertaken to establish whether or not interventions produce their intended effects (the extent to which a program causes change in the desired direction) Basic aim is to produce an estimate of the “ net effects” of an intervention free of the influence of other processes and events that may produce the same results [establishing causality]. Comparison of outcomes in targets who have experienced an intervention and those who have not is the underlying common frame of reference for all impact evaluations Basic requirements: Well articulated objectives Specification of measures of goal achievement Sufficiently well implemented interventions that there is no question that its critical elements have been delivered to appropriate targets. Data collection plan that allows measures of goal achievement to be derived [commonly quantitative or quantifiable qualitative data]

EFFICIENCY ASSESSMENTS Provide a frame of reference for relating program costs to program results (i.e guages the extent to which a program is producing sufficient benefits to program participants for the costs incurred) Efficiency assessments employ cost-benefit and cost-effectiveness analysis methodologies These can be undertaken before or after a program has been in existence but often only possible after program has been in existence

Cost-benefit analysis Program inputs and program outcomes measured in monetary terms Requires specification, measurement and valuation of all program costs and benefits (costs include both direct and indirect costs of undertaking the program).

Cost-effectiveness analysis Requires program costs in monetary terms and benefits in outcome units Generally less controversial than cost-benefit analysis as regards the need to express outcomes in monetary terms Final measure is expressed as the cost of achieving a magnitude of substantive intervention outcome Amenable to rank ordering of programs in terms of their costs for reaching given goals or the various inputs required for different degrees of goal achievement

PROGRAM UTILITY QUESTIONS Is the programme effective in achieving its intended goals ? Can the results of the program be explained by some alternative process that does not include the programme ? What are the costs to deliver services and benefits to program participants ? Is the programme an efficient use of resources, compared with alternative uses of the resources ?

CONCLUSIONS Monitoring and evaluation are essential elements of the process of implementation of TB/HIV programs (as for other public health and social programs) Program implementation and utility with particular focus on coverage, delivery, impact and cost-effectiveness of interventions constitute the cardinal areas in monitoring and evaluating TB/HIV activities (and should be planned for at the design stage of intervention programs). Meaningful interpretation of monitoring and evaluation efforts entails specification of services: defining each kind of service in terms of the activities that should take place or in terms of participation by target populations and providers [time, costs, procedures, or a product, i.e. Specific simple program elements that can be identified, and reliably meaningfully counted !]