How frameworks guide program monitoring and evaluation M&E Frameworks How frameworks guide program monitoring and evaluation

Learning Objectives Describe function of Conceptual, Logical and Results frameworks Apply Proximate Determinants Framework to M&E Planning Use logical frameworks to guide M&E of HIV programs Apply achievements to results frameworks

Session Overview Function of frameworks Conceptual Frameworks Proximate Determinants Logic Frameworks PMTCT program implementation, GFATM Results Frameworks PEPFAR GFATM M&E Reporting Framework Activity Developing a logic framework for an HIV program

Discussion Why do we use frameworks in M&E?

Functions of Frameworks for M&E Roadmap to program planning, monitoring and evaluation Delineate clear pathways to program goals & objectives Define relationships between program inputs, processes, outputs, and outcomes Describe how program factors interact with external context (environmental factors) Lead to sound implementation and good M&E plans

Types of Frameworks - Conceptual Definition Illustrate relationships among all factors (systemic, organizational, individual) that influence program operation & successful achievement of goals Purpose Show how program components will operate to influence outcomes within wider context Guide identification of indicators Guide impact analysis (causal pathways)

Conceptual Frameworks Makes explicit connections among relevant contextual (environmental) factors and your program Helps clarify the “why” and “how” questions of program operation and design: Assumptions that underlie chosen activities Factors activities are expected to affect Causal linkages leading to achievement of program objectives

Conceptual Framework Examples: Proximate Determinants Premise of “proximate determinants” frameworks: Factors leading to health or demographic outcome operate through an ordered path of determinants Determinants grouped in proximity to outcome Underlying (socioeconomic) factors farthest away, biological closest “Proximate” determinants take intermediate position between underlying and biological Proximate determinants are affected by underlying factors & affect the biological determinants Proximate determinants are what most programs seek to influence

Proximate Determinants Example: HIV (STI) Incidence1 Underlying determinants Socioeconomic and socio-cultural context (personal and community) Health programs environment: VCT, ARV, STD control, condom promotion, IEC Proximate determinants (interventions) Partner acquisition, mixing patterns, concurrency, abstinence Condom use, STI present, risky practices, STI treatment Treatment Biological Determinants Rate of contact between susceptible and infected Efficiency of transmission Duration of activity Outcomes: HIV incidence/mortality, STI incidence 1Boerma and Weir, 2005 1 Boerma and Weir, 2005

Proximate Determinants Framework for HIV/STI1 Underlying Determinants Proximate Biological Health Outcome Demog. Context Sociocultural Socioeconomic Demographic Interventions VCT STI control Condom prom. IEC Blood safety Safe injections Partner acquis. Mixing patterns Concurrency IDU Condom use Other STI Type of sex Needle safety ARV Treatment STI Treatment Exposure of susceptible to infected Efficacy of transmission per contact Duration of infectivity HIV infection Mortality Disease Think about where your programs fit and what proximate determinants you are trying to change 1Boerma and Weir, 2005 1 Boerma and Weir, 2005

Ecological Framework: Links between violence against women and HIV1 Direct transmission Rape Gender & relationship inequality Rape, child sex abuse & IPV Psychological Distress: Chronic Anxiety Depression PTSD Substance use More risky sex: More partners Concurrency Transact. Sex Sex while intox. HIV Indirect transmission More risky male partners: More controlling & violent masculinities More sexual risk taking More likely HIV/STIs Reduced protective powers: More acquiescent femininities More frequent sex Less condom use 1 WHO 2010, Addressing violence against women and HIV AIDS: What works?

Types of Frameworks -Logic Definition Pictorial diagram that summarizes essential program information by describing its main elements (inputs, activities, outputs, outcomes) & how they work together to affect a desired health outcome Purpose Make intended program outcomes explicit Show internal logic of program Reveal assumptions of how program leads to outcomes Guide indicator selection Promote communication among stakeholders Guide impact evaluation Source: CDC Evaluation Guidance Handbook at: http://www.cdc.gov/hiv/aboutdhap/perb/guidance/chapter4.htm

Logic Frameworks - Components Problem statement The “why” of program implementation Statement what puts a population at risk for HIV Inputs Program resources like staff, curricula, money Activities What the program does like outreach, training, testing Outputs Deliverables of program like people trained or tested Outcomes Program results like change in knowledge, service use Impact Long term change in health status like decreased HIV incidence Source: CDC Evaluation Guidance Handbook at: http://www.cdc.gov/hiv/aboutdhap/perb/guidance/chapter4.htm

Logic Framework Example: PMTCT Program1 Problem Statement HIV rates rising in pregnant women & infants in Thailand. Transmission risk from mom to child significant during pregnancy & delivery, with additional risk pp via breastfeeding Inputs Staff, funding from CDC GAP & gov’t, PMTCT counseling/protocol, HIV test kits, ARV drugs, breast milk substitutes Activities ANC services include counseling, testing & referral & referral, distribute ARV prophylaxis, provide infant feeding counseling & support 1CDC M&E Guide, Version 1, Appendix 1

Logic Framework Example: PMTCT Program, cont’d.1 Outputs (per quarter) 500 women participate in a 2 hr. ANC session 250 of these will get HIV test results All (of 500) who test positive will be referred for ARV prophylaxis, infant feeding C&S Half of all positive women will receive prophylaxis and infant feeding counseling and breast milks substitutes Outcomes Increase in access to HIV testing Increase in knowledge of status Increase in knowledge about prophylaxis Increase in ARV prophylaxis use by pregnant HIV+ Increase in rapid infant weaning/use breast milk substitutes Impact: ↓ HIV incidence in infants, ↓ morbidity/mortality among people living with HIV/AIDS 1CDC M&E Guide, Version 1, Appendix 1

Developing a Logic Model1 Input from all levels of stakeholders Program staff, program participants and others to analyze intended program outcomes A good problem statement leads to a good logic model Analyze the problem, program resources, desired outputs, outcomes and impact Design activities to achieve outputs and outcomes Usually expressed in diagram or table 1CDC, ETA Evaluation Briefs, Logic models, No. 2, Dec. 2005

Types of Frameworks – Results1 USAID funded projects, like a logic framework Definition A tabular/graphic description of a program’s strategy for achieving a specific strategic objective (SO) through intermediate results, with a narrative supplement. Purpose Like a logic framework, planning & management tool for programs Specifies out exactly how, through a set of intermediate results, the program SO will be achieved Provides a basis of consensus for all program stakeholders Conveys the development hypothesis—plausible linkages between the intermediate results and SO 1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947

Results Frameworks - Components1 Strategic Objective The ultimate end point of the Results Framework—the goal of the program “Standard by which the operational unit (performance) is willing to be judged” Appropriate level of ambition means the SO should be achievable, clear and measurable Intermediate results Achievable in the short term, lower level than SO An intermediate step leading to achieving the SO Lower level of impact than the SO; cumulative effect leads to the SO 1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947

Results Frameworks - Components1 Clarification of causal linkages Narrative that shows how intermediate results are linked to the SO Links can be between results which then leads to SO Critical Assumptions Set of conditions necessary under which the development hypothesis (strategy for achieving the SO) will hold true Not under project/USAID control, but will affect SO/results 1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947

Results Framework Example – PEPFAR funded ART Program SO: Utilization of ART services IR-1: Availability of quality services IR-2: Demand for services IR-1.1: Increase ART sites IR-2.1: Increase knowledge of ART IR-1.2: Supply sites with ARVs IR-2.2: Increase referral from VCT IR-1.3: Training for providers

Developing a Results Framework Set an appropriate strategic objective Identify intermediate results Clarify causal linkages between results Identify critical assumptions Spend time on the narrative portion so that all stakeholders understand What is trying to be achieved How to report verifiable achievements

GFATM M&E Framework1 Key Principles Small set of indicators Country ownership, harmonization, consistency, balance between routine & survey data, avoid duplication of efforts Small set of indicators Focus on multi-agency M&E Toolkit (http://www.theglobalfund.org/en/performance/monitoring_evaluation/) 12-18 indicators per grant Relies on UNAIDS MERG documents/indicators No “Global Fund” indicators (country ownership) Mix of short- & long-term indicators Quarterly or 6-monthly for some 15-18months for others, important for next Phase 1The Global Fund Monitoring and Evaluation KEY MESSAGES, http://www.theglobalfund.org/en/files/about/structures/lfa/background/LFATrainingMaterials/ME/ME_Key_Principles.pdf

GFATM M&E Framework, contd.1 Avoid duplication Use existing in-country indicators, data systems, planned surveys Strengthen existing systems and budget for data gaps Indicator targets need to match grant proposals Existing tools for performance evaluation M&E Toolkit, M&E Systems Strengthening Tool, Data Quality Assessment Tool Tools developed for GFATM grants, but applicable to many contexts, as we’ll see 1The Global Fund Monitoring and Evaluation KEY MESSAGES, http://www.theglobalfund.org/en/files/about/structures/lfa/background/LFATrainingMaterials/ME/ME_Key_Principles.pdf

New Donor Demands: Gender PEPFAR, UNAIDS, GFATM, WB require gender integration into programs Addressing gender in meaningful way = measureable All agencies have strategies to address gender equality Many resources out there (we’ll examine some at the end of the week)

Activity: Developing a Logic Framework Assemble workshop group-work members Use provided CDC-GAP activity sheet: “Logic Model Worksheet for Planned Implementation and Outcomes” Take about an hour to complete sheet Resulting logic models to be presented back (5 minutes for each group) by group representative This draft logic model will guide your group work and form the basis for indicator selection; you will have the opportunity to revise based on feedback

MEASURE Evaluation is a MEASURE project funded by the U.S. Agency for International Development and implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. Views expressed in this presentation do not necessarily reflect the views of USAID or the U.S. Government. MEASURE Evaluation is the USAID Global Health Bureau's primary vehicle for supporting improvements in monitoring and evaluation in population, health and nutrition worldwide.