Training Modules: Monitoring and Evaluation of Population, Health and Environment (PHE) Programs MEASURE Evaluation [COMMENTS, NOTES, AND INSTRUCTIONS.

Training Modules: Monitoring and Evaluation of Population, Health and Environment (PHE) Programs
MEASURE Evaluation [COMMENTS, NOTES, AND INSTRUCTIONS FOR THE FACILITATOR ARE WRITTEN IN ALL CAPS AND BRACKETS [] THROUGHOUT THE POWERPOINT PRESENTATION AND THE FACILITATOR NOTES. SUGGESTED TEXT FOR YOU TO READ, ALOUD, TO PARTICIPANTS ARE WRITTEN IN SENTENCE CASE AND NOT INCLUDED IN BRACKETS.] [HINTS FOR ADAPTATION: THESE TRAINING MODULES AND FACILITATOR NOTES ARE WRITTEN FOR PEOPLE-CENTERED PROGRAMS WITH A FOCUS ON HEALTH, POPULATION, AND ENVIRONMENT INTERVENTIONS. THE MONITORING AND EVALUATION (M&E) KNOWLEDGE AND SKILLS USED IN THESE MATERIALS IS LARGELY BORROWED FROM THE HEALTH AND POPULATION SECTORS AND APPLIED TO AN INTEGRATED PROGRAMMING CONTEXT TO REACH A BROADER AUDIENCE, SPECIFICALLY THOSE WHO WORK IN INTEGRATED COMMUNITY DEVELOPMENT PROGRAMS IN DEVELOPING COUNTRIES. BECAUSE THESE PROGRAMS ARE DIVERSE AND REFLECT WIDE VARIATION IN APPROACH, TARGET BEHAVIORS, COUNTRY-CONTEXT, AND INTENDED OUTCOMES, NO SINGLE M&E GUIDE CAN FILL ALL M&E KNOWLEDGE AND PRACTICE GAPS. THEREFORE, THROUGHOUT THE TRAINING MODULES, FACILITATOR NOTES, AND TOOLKIT MATERIALS, YOU WILL LIKELY NEED TO MAKE CHANGES TO REFLECT YOUR SPECIFIC PROGRAMMATIC CONTEXT, THE COUNTRY IN WHICH YOU WORK, THE SKILL LEVEL OF YOUR AUDIENCE, AND YOUR SECTOR-SPECIFIC REPORTING NEEDS. THE TRAINING MODULES AND TOOLKIT MATERIALS ARE INTENDED FOR YOU TO USE AND EMPLOY AS YOU SEE FIT – TAKE THEM AND MAKE THEM YOUR OWN. ADD, DELETE, ALTER, AND REORDER THE SLIDES TO BEST MEET YOUR NEEDS.] [FOR SIMPLICITY AND CONSISTENCY, THE NOTES FOR THIS COURSE ARE WRITTEN USING THE PROGRAMMATIC TERMINOLOGY AND APPROACH, “POPULATION, HEALTH AND ENVIRONMENT” (PHE). ALTHOUGH “PHE” IS USED THROUGHOUT THE SLIDES, THE TRAINING MODULES AND TOOLKIT MATERIALS ARE INTENDED TO REACH A WIDER AUDIENCE OF PEOPLE AND PROGRAMS WHO WORK TO IMPROVE THE HEALTH AND WELLNESS OF POPULATIONS AND ENVIRONMENTS THROUGH COMMUNITY-BASED APPROACHES IN DEVELOPING COUNTRIES. YOU WILL NEED TO MAKE THE LANGUAGE CONSISTENT AND UNDERSTANDABLE FOR YOUR TARGET AUDIENCE. IF YOU WANT TO REFER TO YOUR PROGRAMS AS “INTEGRATED DEVELOPMENT PROGRAMS” OR “COMMUNITY-BASED PROGRAMS”, MAKE THE CHANGES AS APPROPRIATE. SIMILARLY, IF YOU WISH TO CHANGE THE FOCUS OF THE EXAMPLES TO INCOME GENERATION, GENDER, CONSERVATION, EDUCATION, ETC., FEEL FREE TO MAKE THOSE ADAPTATIONS. TRY BE CONSISTENT THROUGHOUT.] [LASTLY, BEFORE YOU IMPLEMENT THESE TRAINING MODULES IN WHOLE OR IN PARTS, DO A PRE-TEST TO DETERMINE THE BASE LEVEL OF YOUR PARTICIPANTS. AN EXAMPLE PRE-TEST IS INCLUDED IN THE TOOLKIT MATERIALS IN THE “ADDITIONAL TRAINING MATERIALS” FOLDER: Example participant pre-test assessment of knowledge PHE ). WHEN YOU FINISH, HAVE THE SAME PARTICIPANTS COMPLETE A POST-TEST TO DETERMINE IF THEY LEARNED THE SKILLS AND KNOWLEDGE YOU INTENDED. EXAMPLE POST-TESTS ARE ALSO PROVIDED.] Developed by Caryl Feldacker. 1

Agenda [FILL IN YOUR TRAINING MODULE AGENDA AND TIMING HERE. THE TIMING IS FLEXIBLE. YOU COULD COMPLETE ALL TRAINING MODULES AND ACTIVITIES IN AS LITTLE AS THREE FULL DAYS OR YOU MIGHT SPREAD THE LESSONS AND ACTIVITIES OUT OVER SEVERAL MORNINGS, DAYS, OR WEEKS DEPENDING ON YOUR NEEDS AND TIME CONSIDERATIONS. YOU MAY ALSO IMPLEMENT ONLY SPECIFIC MODULES, DEPENDING ON YOUR TIME CONSTRAINTS, THE SKILL OF THE PARTICIPANTS, AND YOUR INTENDED TRAINING OUTCOMES.] 2

Training Modules Introduction to PHE Frameworks, Part 1 – Conceptual
Introduction to M&E Frameworks, Part 2 – Logic models Indicators Frameworks, Part 3 – Indicator matrixes Assessment and evaluation design [YOU MAY CHOOSE TO COMPLETE ALL MODULES, SPECIFIC MODULES, SPECIFIC SLIDES OR REARRANGE THE MODULES TO FIT YOUR SPECIFIC NEEDS. YOU MAY ALSO ADD TRAINING MODULES AS NEEDED/DESIRED BY YOUR SPECIFIC ORGANIZATION AND TRAINING AUDIENCE.]

Starting on the Same Page…
First, everyone should complete this free, online, two-hour course: Then, all participants should be able to: Identify the basic purposes and scope of M&E Identify the basic criteria for the selection of indicators Describe how indicators are linked to frameworks Identify types of data sources Describe how information can be used for decision making [THE FACILITATOR AS WELL AS ALL PARTICIPANTS SHOULD COMPLETE THIS FREE TRAINING COURSE IN PREPARATION FOR THIS TRAINING. IT WILL MAKE SURE EVERYONE STARTS WITH THE SAME UNDERSTANDING OF THE KEY TERMS USED IN THESE TRAINING MATERIALS. IF THE PARTICIPANTS DO NOT HAVE REGULAR, CONSISTENT ACCESS TO THE INTERNET, THE COURSE IS INCLUDED AS A PDF IN THE TOOLKIT FOLDER: ADDITIONAL TRAINING MATERIALS: MEASURE Evaluation M&E Fundamentals course.pdf] 4

Module 1: Training Overview
[MATERIALS FOR THE MODULE: 1) HANDOUTS OF THE SLIDES, AS NECESSARY; 2) COPIES FOR ALL PARTICIPANTS OF THE MEASURE EVALUATION PUBLICATION, “A GUIDE FOR MONITORING AND EVALUATING POPULATION-HEALTH-ENVIRONMENT PROGRAMS”. The PDF IS INCLUDED IN THE TOOLKIT ADDITIONAL TRAINING MATERIALS FOLDER AND IS AVAILABLE ONLINE HERE: ]

Training Programmatic Focus
This course is geared for programs that: Involve active partnerships between communities, non-government organizations, or government entities; Promote an integrated, community-based approach to problem solving; Operate in limited resource settings (smaller budgets); Collaborate between population, health and environment (PHE) sectors by implementing activities in multiple programmatic areas including, but not limited to: family planning, education, income generation, natural resource management, gender, sanitation, or agriculture. [READ SLIDE] It is important to understand that calling these programs “Population, Health and Environment, PHE, programs” does not mean that other programming components are not important. What we call these programs is not the most important factor. You may refer to these programs by the broader term, “community development” or “integrated development.” What is critical to understand, and the focus of this course, is that these types of programs simultaneously address multiple issues that affect communities, mostly in rural areas. The important part is understanding that PHE, in all its forms and by whatever name, is about cross-cutting activities and integration across sectors.

Training Target Audience
This training course and its modules are most useful for: Organizations that are familiar with PHE programs and the PHE approach Staff who are involved with monitoring and evaluation in an integrated programming context Managers and staff who design, plan, and implement PHE programs Consultants and trainers of technical staff in integrated PHE programs [YOU MIGHT NEED TO ADAPT THE MODULES TO BETTER FIT YOUR AUDIENCE NEEDS]

Training Objectives To increase understanding of the basic concepts and practical approaches for performance monitoring and evaluation for integrated population, health and environment programs To learn state-of-the-art tools and techniques used specifically to monitor and evaluate population, health and environment programs To gain hands-on experience in designing components of monitoring and evaluation plans for population, health and environment programs [READ SLIDE] Some of you will find this to be a review of things you already know. Some of you will find this all to be new and are learning through this course, while others of you will find it just right, helping you solidify skills you already have. Whatever your skill level, there are many activities where you will be able to work at your own level for more flexibility.

Training Learning Methods
Lectures In-class exercises Small group work Group presentations [NOTE: YOU WILL NEED TO ADAPT THESE NOTES TO THE COMPONENTS AND TIMING OF YOUR SPECIFIC COURSE. YOU, AS THE TRAINER, WILL ALSO HAVE TO DETERMINE WHEN AND HOW YOU WILL ORGANIZE THE GROUP WORK COMPONENTS. WILL PARTICIPANTS CHOOSE THEIR OWN GROUPS AND WORK ON THEIR OWN MATERIALS OR WILL YOU ORGANIZE GROUPS AND TASKS FOR THE PARTICIPANTS? AT THE END OF EACH MODULE, ACTIVITY, OR DAY, AS WELL AS AT THE END OF THE COURSE, PARTICIPANTS SHOULD HAVE A CHANCE TO PRESENT AND GAIN FEEDBACK FROM PEERS – A CRITICAL COMPONENT OF M&E.]

LAYING THE FOUNDATION Before we begin, let’s create a common understanding of what we mean by PHE. [AGAIN, IF YOU WISH TO USE PHRASES SUCH AS “INTEGRATED DEVELOPMENT,” “COMMUNITY DEVELOPMENT,” OR “CROSS SECTORAL PROGRAMS,” ETC., FEEL FREE TO CHANGE OR MODIFY THIS LANGUAGE AND THE SUBSEQUENT SLIDES. BE CONSISTENT].

What is PHE? DEFINITION:
A community development model that links population, health and environment (PHE) sectors and supports cross-sectoral collaboration and coordination, particularly in biodiversity-rich areas (BALANCED: Within one or more communities, these programs actively link environmental activities with activities that seek to improve reproductive health, always including, but not limited to, provision of family planning services (Robert Engelman, Worldwatch Institute) [READ SLIDE] How would you define PHE? [HERE YOU COULD MAKE A LIST OR STATEMENT OF WHAT PHE LOOKS LIKE WITHIN YOUR ORGANIZATION. (FOR EXAMPLE: REFORESTATION EFFORTS + COMMUNITY BASED DISTRIBUTION OF CONTRACEPTIVES) AND APPLY IT FOR YOUR COURSE] [These slides are taken from : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at ]

What Does PHE Stand for? P = Reproductive health, family planning H = Water, sanitation, malaria, vaccines, HIV/AIDS, MCH E = Natural resource management, biodiversity conservation, fisheries management, eco-agriculture Amenable to other sectors +E = Education (non-formal & formal) Energy (fuel-efficient smokeless stoves, biogas) Eco-tourism, sustainable tourism development Enterprise/livelihood development/integrated development Perhaps one of the most important questions is ‘what exactly is PHE?’ Obviously the letters themselves mean population, health and environment. As basic pillars, they include: P = Reproductive health & family planning H = Water, sanitation, malaria, HIV/AIDS, MCH E = Natural resource management, biodiversity conservation, fisheries management, eco-agriculture However, PHE is also amenable to other sectors and takes into consideration the social, cultural, and livelihood aspects of the community. Population, health and environment then combine to create PHE – which can be defined as “the linkage, within a community or group of communities, of natural resource management or similar environmental activities and the improvement of reproductive health, always including but not limited to provision of family planning services” (Bob Engelman). [These slides are taken from : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at ]

Why Integrate? Social and environmental factors linked to food insecurity in coastal Philippines Now that we know what it is, the question is, why should we integrate these three sectors? [LOOK THROUGH THE SLIDE AND POINT OUT COMPONENTS OF THE FIGURE THAT MIGHT APPLY TO YOUR PROGRAM]. In your own experience, what makes you consider or apply an integrated model? [IF NO ONE SPEAKS UP, HAVE THEM LOOK AT THIS SITUATIONAL DIAGRAM OF A COASTAL AREA IN A DEVELOPING COUNTRY AND DISCUSS HOW P, H, AND E ISSUES ARE RELATED] [These slides are taken from : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at ]

Reasons for Integration
Integrated lives Address root causes Enhance relationship Access different communities Economic efficiencies Build on existing frameworks Broader community participation Value added from integration Why do we integrate? [IF YOU HAD A GREAT DISCUSSION ON THE PREVIOUS SLIDE, YOU MAY WISH TO SKIP SOME OF THESE DETAILS] First, intuitively, it makes sense to integrate. We lead integrated lives – water quality affects our health for example. For the same reason, projects that integrate sectors also allow organizations to address the root cause of the situation in which they are working, rather than just proximate causes. For example, as seen in the previous slide, in many areas, while there are immediate threats to the biodiversity, such as habitat conversion, the underlying driver for it is high population growth. You can’t achieve long term change if you only address the immediate threats. For the organizations involved there are several potential benefits. Conservation organizations may be able to develop better relationships with communities by integrating a health component – which allows them to provide something tangible for the communities. For health organizations, linkage with natural resource management has enabled access to communities and clients that otherwise would be impractical or too expensive to reach. And by combining resources, organizations can potentially implement their projects in a more efficient manner, sharing transportation and field staff expenses with other organizations. To support the economic advantage of these efforts, PATH Foundation Philippines, implementers of the Integrated Population and Coastal Resource Management (IPOPCORM) project in the Philippines, recently completed a large scale operations research program which found that their projects were much more economically efficient than trying to deliver FP and conservation services independently. [MORE INFORMATION ON THIS PROJECT IS AVAILABLE IN THE DOCUMENT “Additional training resources and references” FOUND IN THE ADDITIONAL TRAINING MATERIALS FOLDER AND HERE: ] Integrated projects can work with a wider variety of frameworks, allowing them a greater chance of building on existing agendas and tapping into available resources. Approaching the project in an integrated fashion may also affect the manner in which the community participates. Integration generates active involvement of a broader segment of the community and increases women’s and youth’s participation in resource management and men’s involvement in reproductive health. Finally, there may be a synergistic benefit derived from integrating sectors – that working together across PHE sectors, and implementing complementing activities and strategies, may create momentum that yields surplus or extra, positive, outcomes outside of the intended intervention outcomes. The “value-added” effects and results of these programs will be addressed throughout the training and discussed in more detail in later modules. Which of these reasons for integration has been most relevant and significant for you and why? [THESE SLIDES ARE TAKEN FROM : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at ]

Synergy from Combination
Social systems & ecosystems are interrelated and linked Whatever effects one system, affects the other Small improvements in either system reinforce each other To turn around both systems from degradation to health PHE is grounded in basic concepts of ecology and the belief that ecosystems and human social systems are inter-related and form a unified whole (linked eco-social system). As such, whatever affects one system affects the other. Small improvements in either system reinforce each other and can turn around both systems from degradation to health [SYNERGY CONCEPT TAKEN FROM MARTEN, G. ENVIRONMENTAL TIPPING POINTS: HOW TO RESTORE OUR NATURAL SECURITY. UNPUBLISHED MANUSCRIPT ] [These slides are taken from : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at ]

Types of Integration Parallel: projects conducted in a single area without coordination among organizations Coordinated: projects conducted in the same area with some level of coordination Cross-sectoral: projects involving several sectors that are conceptually linked Integrated: projects involving several sectors that are conceptually and operationally linked In general, there are four types of integration. [READ SLIDE] Parallel: Projects that follow traditional, single-sector planning and implementation phases. Coordinated: Projects being conducted in the same area, where some level of coordination exists among the organizations. They are not necessarily formally working on a project together but rather coordinating efforts on separate projects. Cross-sectoral – Projects that involve several sectors, though they may not fully integrate their efforts. It aims to link the sectors at least conceptually, but may not link them operationally. For example, the Madagascar “Nature, Health, Wealth and Power” framework presents a platform through which USAID is building bridges among sectors and programs that address aspects of Environment (Nature), Health, Livelihood (Wealth) and Governance (Power). Integrated - projects involving several sectors that conceptually and operationally linked e.g., IPOPCORM. [ASK PARTICIPANTS FOR SOME EXAMPLES OF PROJECTS THEY HAVE IMPLEMENTED (OR HAVE OBSERVED) THAT FIT ONE OR MORE OF THESE TYPES. DRAW ON THEIR EXAMPLES UNTIL ALL APPROACHES HAVE BEEN ILLUSTRATED OR GROUNDED IN CONCRETE EXAMPLES. UNDERSCORE THE RATIONALE FOR WORKING TOWARD FULLY-INTEGRATED PROGRAMS OVER THE PARALLEL AND OTHER LESS-INTEGRATED MODELS. THIS WORKSHOP WILL EMPHASIZE THE FULLY-INTEGRATED APPROACH.] [These slides are taken from : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at ]

Achieving Integration
Conceptual Integration Field Integration Community Involvement PHE Partnerships Mobilizing Resources Planning Ahead for Sustainability [YOU WILL NEED TO ALTER THIS LANGUAGE TO ADDRESS THE SPECIFIC WAY YOUR ORGANIZATION OR PROGRAM ADDRESSES AND IMPLEMENTS AN INTEGRATED APPROACH. BELOW IS SOME LANGUAGE THAT YOU COULD CHANGE DEPENDING ON THE FOCUS OF YOUR PROGRAM, ITS HISTORY, AND ITS PLANNED FUTURE/SUSTAINABILITY.] One of the most important steps in creating a PHE project is to conceptually link P, H, and E – to each other, and to the outcomes an organization is trying to achieve. This means making sure that your field programs are linked. This linkage could be as easy as making simultaneous visits to the communities by field staff from various sectors to cross training staff – for example by making sure your natural resource staff are trained in family planning methods or that your health staff understand the importance of reforestation. Or, your outreach to communities could include messages about the linkages between P-H-E. Keep in mind that your PHE partnerships could be formed at the community, district, organization, or technical levels, building bridges between entities. Furthermore, proposals and funding solicitations could be made to entities that focus on multiple sectors, or resources may need to come from traditional single-sector funders to be used in cooperation with efforts in other sectors. By working across sectors, you may be able to cut costs by reducing redundancies – joint field visits in one vehicle, joint monitoring efforts, joint community meetings, staff cross-training. In this way, you may be able to work at a higher level with fewer resources. Lastly, planning ahead is important as you consider how you may be able to work across sectors at various points in the life of a project, anticipating community needs that may go beyond the original or intended scope of the project. Planning ahead helps you consider how your programmatic efforts will be sustained and maintained after your funding or implementation cycle. Who will assume ownership? How will the community take over? How will you build capacity in your partners? What examples can you share from your own countries/organizations of integrative mechanisms either in PHE or other sectors i.e., examples from integrated rural development-agriculture projects etc.? [NOTE: AT THIS POINT, YOU MAY WISH TO DO AN INTEGRATION ACTIVITY SUCH AS THE ONE ASSOCIATED WITH THIS GROUP OF SLIDES ON PHE. AN INTEGRATION ACTIVITY CAN BE FOUND AT : Session 2: Commonalities, Conditions & Characteristics of PHE Projects, slide 19]

“Valuing” Gender as Integral to PHE
Consideration of gender begins with attention to two central questions: How will gender relations (relationships between men and women) affect the achievement of sustainable results? How will my program affect, influence, or change the relative status of men and women? (Will it reduce differences in men’s and women’s status? Will it help create equality between men and women?) As part of the PHE approach, it is important not only to collaborate across sectors but to reach out to include all members of the communities in which you work. It is also critical that that try to ensure that women play an equal role in the development, implementation, and benefits of our programs. Therefore, PHE programs should set and maintain a positive example by promoting the inclusion of women in all aspects of society, including involvement in population, health and environment programming and policy. This means including women as decision makers, participants, and leaders throughout the program design, planning, and implementation stages. “Integrating gender” means thinking about gender starting at the program design phase and carrying through to the program completion – seeking, including, solidifying women’s active engagement at all levels and stages of the program. Gender is a good example of a value added component of integrated PHE programs. For example, in many PHE efforts, our programs reach out to both men and women in conservation decisions or in programs aimed at livelihoods development. By seeking the active and equal participation of all members of the household, including men and women [AND POSSIBLY YOUTH IF THIS IS YOUR PROGRAMMATIC FOCUS], it is hoped that shared decision making in these example conservation activities could translate to a more equitable voice for both men and women in health and family planning decisions as well as in society in general. To help us all consider how our integrated programmatic and policy interventions will influence women, and potentially add value to our programs, let’s think about these two questions from the Inter-Agency Gender Working Group of USAID: [READ SLIDE] We’ll revisit issues related to value added and integrating gender later in the course. [IN ADDITION TO ASKING THESE QUESTIONS, YOU MIGHT CONSIDER HAVING AN ACTIVITY/DISCUSSION ABOUT THE PARTICULAR CULTURAL AND SOCIAL CONTEXT IN WHICH PARTICIPANTS LIVE AND WORK, INCLUDING HOW GENDER AFFECTS RELATIONSHIPS, OPPORTUNITIES, AND EXPERIENCES IN THEIR DIVERSE COMMUNITIES. AUGMENT THE DEPTH OF THE GENDER DISCUSSION TO MEET YOUR NEEDS.] [NOTE: THESE QUESTIONS COME FROM THE ADS ( of USAID. IF YOU WOULD LIKE TO FOCUS MORE ON GENDER, THERE ARE A NUMBER OF ANALYTICAL TOOLS TO HELP YOU IN THIS PROCESS. ONE SOURCE OF INFORMATION IS HERE: MORE INFORMATION ABOUT GENDER IS AVAILABLE IN THE TOOLKIT: ADDITIONAL TRAINING RESOURCES FOLDER, “Additional training resources and references”.] 18

Module 2 Frameworks, Part 1 Conceptual frameworks
[MATERIALS FOR THE MODULE: 1) HANDOUTS OF THE SLIDES, AS NECESSARY; 2) COPIES FOR ALL PARTICIPANTS OF THE MEASURE EVALUATION PUBLICATION, “A GUIDE FOR MONITORING AND EVALUATING POPULATION-HEALTH-ENVIRONMENT PROGRAMS (The PDF of the guide is online here: ) AND ATTACHED IN THE ADDITIONAL TRAINING MATERIALS; 3) LARGE PIECES OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP, FOR ACTIVITY; 4) THICK MARKING PENS, ONE OR TWO PER GROUP; 5) POST-IT NOTES.]

Module Outline Introduction to frameworks Conceptual frameworks
Including goals and objectives Logic models Indicator matrixes Exercises developing frameworks Linking frameworks and M&E planning Work on group project [READ SLIDE] Frameworks come in many forms and vary widely depending on the programmatic sector, donor, academic institution, etc. This course is going to present three general frameworks frequently used in the population and health sectors. The use of these three frameworks does not indicate that these frameworks are the best for all settings or organizations or that other frameworks are wrong. However, these three frameworks are excellent tools to guide you from program design to program planning through to monitoring and evaluating your program. You may alter these frameworks to better suit your needs or modify them for your specific context. In the training course, we will discuss conceptual frameworks, logic models, and indicator matrixes. After finishing all the modules, you will know each type of framework in terms of its characteristics and uses, and you will get to practice developing these frameworks. For PHE, and the complexity of these integrated efforts, drafting all three types of frameworks as part of your overall planning and M&E efforts can help you consider and demonstrate the complex linkages between sectors that you address through your program. To design and use all three frameworks well, it is important to know the differences between the types and how they are used. In this module, we are going to talk about conceptual frameworks. These frameworks are used to describe and draw the macro-level or larger scale factors or issues that influence a health or conservation problem, helping us to understand a problem we wish to address. As part of this session, we will also talk about goals and objectives. Again, keep in mind that many different organizations, people, and donors call these frameworks by different names and use different formats. The frameworks used in the course are meant to be examples and are not intended to be interpreted as the “correct” framework for all programs. The ideas behind them, the development process, and their utilization for M&E are most important, so let’s not get too bogged down in the particulars of the names or formats of any of these models. In the training sessions and activities for all three frameworks, I am most interested in how you create, explain, and apply your framework and in your understanding of how these models can help you clarify and explain your thoughts, assumptions, and relationships for M&E.

Brief Summary of Three Frameworks
Type of Framework Brief Description Use for Program Management Basis for Monitoring and Evaluation of PHE Conceptual Shows the interactions of various factors that operate in your program environment Determine which factors the program will influence and how Helps explain and promote the multi-sector influences and integrated approach Logic model Logically links inputs, processes, outputs, and outcomes of planned activities Shows the causal relationship between inputs and reaching the objectives Shows intended progress of activities at all stages of the program – aids indicator selection Indicator matrix Logically links program goal, objectives, outputs, outcomes, and activities Shows the causal relationship and steps between activities and meeting objectives Guides monitoring and aids assessment by stating the information you will collect, how, when, where and why. How do the frameworks fit together? Before we look at the first framework, the conceptual framework, let’s talk briefly about what these frameworks are and how the frameworks build upon, and complement, one another in the context of M&E. [READ SLIDE] These three frameworks are used to inform programming and for monitoring and evaluation of the program from A to Z. Drafting these frameworks, especially in a collaborative effort with partners, also encourages implementation of and adherence to M&E plans. Linking the frameworks makes sense. First, the conceptual framework puts the PHE problem into its greater societal or community context [FOR THE CONSERVATION OR ENVIRONMENT SECTOR, RESULTS CHAINS OR THREAT ANALYSIS ALSO DEMONSTRATE THIS CONTEXTUAL UNDERSTANDING]. Second, logic models help to show the logical connections between the inputs and processes and outputs of planned activities – helping you link your activities to the program’s objectives (shorter term outcomes) and goals (longer term impact). Lastly, a indicator matrix is used to put it all together for M&E planning. The indicator matrix, often in spreadsheet or table format, lays out the programmatic steps that you will take to achieve your objectives, the ways that you will measure progress in both the short- and long-term, the methods you will use to collect this information, and the way that this information could be used to improve your program. By using all three frameworks, you will have a guide to move from program planning, to implementation, to final assessment. The process of designing these frameworks is useful to clarify project and partner understandings of the key assumptions and other necessary elements for a successful intervention. It is also essential to develop an appropriate and useful M&E plan. Without developing a consensus on the foundations and context of the program and its success, monitoring and evaluation cannot be managed in a coherent or comprehensive way in order to contribute to effective management and the achievement of that success.

Type of Framework Brief Description Use for Program Management Basis for Monitoring and Evaluation of PHE Conceptual Shows the interactions of various factors that operate in your program environment Determine which factors the program will influence and how Helps explain and promote the multi-sector influences and integrated approach Logic model Logically links inputs, processes, outputs, and outcomes of planned activities Shows the causal relationship between inputs and reaching the objectives Shows intended progress of activities at all stages of the program – aids indicator selection Indicator matrix Logically links program goal, objectives, outputs, outcomes, and activities Shows the causal relationship and steps between activities and meeting objectives Guides monitoring and aids assessment by stating the information you will collect, how, when, where, and why. Lets start with the conceptual framework.

Module Learning Objectives
At the end of the session, participants will be able to: Explain why conceptual frameworks are helpful for PHE programs Draw a conceptual framework for a specific program Design goals and objectives for specific intervention programs to match the conceptual framework [READ SLIDE]. Conceptual frameworks form the basis for program development and rationale. As part of this module, we will discuss how these models are helpful for PHE and the value of creating a conceptual framework. Participants will be expected to design a framework for a specific intervention program and later use this framework for developing logic models and an indicator matrix. Stemming from the conceptual model exercise, another learning objective of this module is to write quality goals and objectives. Goals and objectives are the core of any M&E system. If the program goals and objectives are written in such a way that they can be easily distinguished from each other and measured, the job of M&E will be much easier. Unfortunately, many times, this is not the case. In this section, we will learn how to write program goals and objectives so that they are easily monitored and evaluated.

Conceptual Frameworks
Definition: Diagram that identifies and illustrates the relationships between all relevant systemic, organizational, individual, or other salient factors that may influence program/project operation and the successful achievement of program or project goals. Purpose for M&E: To show where the program fits into wider context To clarify assumptions about causal relationships To suggest causal pathways [READ SLIDE]. Conceptual frameworks are sometimes called “research” or “theoretical” frameworks. A conceptual framework is a useful tool for identifying and illustrating a wide variety of factors and relationships that may affect program success. Conceptual frameworks take a broad view of the program itself in order to clarify the relationship of its activities and its main goals to the context in which it operates. The design of the conceptual framework should show the interrelationships between all factors that are relevant to achieving the program’s goals. These factors can be systems, organizations, government or institutional policies, infrastructure, population characteristics, or other features of the operational landscape that may help or hinder the program’s success. Constructing a conceptual framework clarifies the complete context affecting the outcome of an program or intervention. Constructing a conceptual framework also shows your assumptions about the causal relationships between significant features of the program context, clarifying aspects that your planned intervention may affect as well as other factors beyond your control. Identifying the variables that factor into program performance, and organizing the explicit ways they interact with each other, sets the stage for defining the objectives you can reasonably expect from your program activities. Clarifying all of these issues is a critical step toward designing valid measures for analyzing, or evaluating, the success of those interventions. A major caution is that these frameworks do NOT show the relationships that your program will affect. This shows the underlying causes, the big picture, and helps people understand the situation you will be addressing. Your program may affect one box, one arrow, or one aspect of this overall issue. It is not to show how you will fix the problem. It is to explain why your program may or may not work. It is to help you show at what level you will intervene and what other factors may affect your programmatic environment. This kind of framework helps you show and communicate the real possibilities and limitations of your program to everyone involved. [AGAIN, DIFFERENT ORGANIZATIONS TEND TO PREFER A SPECIFIC TYPE OF FRAMEWORK. IT IS NOT IMPORTANT TO CONVINCE PARTICIPANTS TO USE ANY PARTICULAR TYPE. THE GIST OF THIS SESSION, HOWEVER, IS THE IMPORTANCE OF DESIGNING A USEFUL FRAMEWORK FOR A SPECIFIC PROJECT IN A SPECIFIC CONTEXT, NOT THE TITLES OR APPEARANCE. IF YOUR ORGANIZATION PREFERS RESULTS CHAINS, THREAT ANALYSIS, ETC., THIS WOULD BE AN EXCELLENT PLACE TO BRING THAT DISCUSSION AND TRAINING INTO THE COURSE.]

Why Are Conceptual Frameworks Useful for M&E of PHE?
To demonstrate knowledge or understanding of the problematic context – situation analysis. To show the linkages between PHE program areas, demonstrating the synergies between integrated problems and integrated solutions. To draw or explain relationships between the problems that a program/project addresses and the external context (environmental factors), demonstrating ways that program activities may affect knowledge, attitudes, skills, and behaviors of the target population. To help identify what evaluation information might be useful to measure program success. In the PHE context, the design of the conceptual framework can help you develop a clear understanding of the specific P, H, and E issues that you wish to address as well as how you will address the problem through an integrated approach. An integrated conceptual framework can help you promote the linkages within the project and give you a format to clearly articulate these synergies during the project planning. Conceptual frameworks are particularly important for PHE programs to show how the linked interventions of population, health and environment sectors will lead to the long-term integrated goal and objectives of the program. The process of designing frameworks also helps to clarify many of the concepts and assumptions underlying basic project activities. It may help you better understand, explain, and promote the integrated nature of the problem and its solution.

Causes of Malnutrition in Society
Child malnutrition, death and disability Outcomes Immediate causes – individual level Inadequate Disease dietary intake Insufficient access to food Inadequate maternal & child care practices Poor water/sanitation & inadequate health services Underlying causes at household/ family level This slide shows an example of a conceptual framework for an integrated nutrition program framework. This framework was provided by UNICEF in 1990 and serves as a guide for assessing and analyzing the causes of child malnutrition. The framework highlights the multi-sectoral nature of the causes of malnutrition, which encompass insufficient access to food, inadequate care for mothers and children, and poor heath services. The causes of malnutrition are also categorized as immediate, underlying, and macro-level. Macro-level causes of malnutrition include inadequate education, the political and ideological superstructure, the economic structure, and the resource base. These factors influence the underlying causes – food, care, and health services – which affect, in turn, the two primary proximate determinants of nutritional status and survival, individual dietary intake and disease. This framework is laid out bottom to top, moving from society-level influences to the individual-level outcome of interest, malnutrition. The orientation is not important – your frameworks could move from left to right or top down. What is important is the clear causal links between all the factors listed. Also, notice the labels on the right side of the slide. These give additional interpretation to the factors listed in the boxes and clarify their proximity to malnutrition and death. [READ THROUGH SLIDE COMPONENTS] Your program would not address all aspects of this conceptual framework. Your program would likely address only one box or circle. The point is that you show how your program fits into the greater context. For example, if your program is about increasing access to food (middle row circle on left side), you would know that your program would influence dietary intake, disease, and then malnutrition. Likewise, you would know that the problem you address is influenced by societal level issues such as quantity of resources. Quantity & quality of actual resources – human, economic and organizational – and the way they are controlled Macro-level causes at societal level Potential resources: environment, technology, people Source: UNICEF, State of the World’s Children, 1998.

Example of a conceptual model for the IPOPCORM project in the Philippines
[POSSIBLE ALTERNATIVE SLIDE FOR ORGANIZATIONS THAT PREFER TO USE RESULTS CHAINS OR THE TYPE OF THREAT IDENTIFICATION IN THIS DIAGRAM. THIS IS THE IPOPCORM MODEL THAT ACCOMPANIES THE SLIDE 13 & 29, AND PROVIDES AN ADDITIONAL EXAMPLE OF A CONCEPTUAL FRAMEWORK OF DRIVERS, THREATS, AND LINKAGES BETWEEN SECTORS. THIS MODEL, ITS DESCRIPTION, AND MORE ON THE METHODOLOGY MAY BE FOUND HERE: OR IN THE SAME DOCUMENT AS A PDF IN THE TOOLKIT: ADDITIONAL TRAINING MATERIALS FOLDER, “PHE_USAID_integrating PHE program manual”]

Result Chain 1: FP intervention to achieve a conservation outcome
Community Based Distribution (CBD) of Contraceptives Increase Access to FP Info and Products Decrease Fertility Fewer Children Couples Have More Time for CRM Activities Habitats Protected Improved Conservation of the Marine Environment Intervention: Community based distribution (CBD) of contraceptives in which community residents are trained and equipped to promote and distribute contraceptives using social marketing approaches. [THIS SLIDE AND NOTES COMES FROM: AND THEY PROVIDE AN ALTERNATIVE FOR THE CONCEPTUAL MODEL DISCUSSION IN THE PREVIOUS SLIDES] Results chains are a series of causal statements that link factors in an “if…then” fashion. Criteria for Good Results Chains: 1. Results Oriented: Boxes should be results you want to see happen (e.g., fertility reduced), not operational steps or activities (e.g., conduct peer education). 2. Causally Linked: Should be clear “if-then” linkages between neighboring boxes. 3. Reasonably Complete: Enough boxes to make logical links but not so many that the chain becomes overly complete or unwieldy. 4. Simple: Only one result per box. 5. Demonstrate Change: Boxes should describe how you expect the relevant factor to increase, decrease, or otherwise change.

Activity: Making a Model
This IPOPCORM drawing should look familiar [FROM SLIDE 13]. It is from the previous section on why we integrate. This picture is a situational diagram of a coastal area in a developing country. In small groups of 4-5 [THIS DOES NOT NEED TO BE THE GROUP FOR THE GROUP PROJECT], I would like you all to take minutes to draw a very rough conceptual framework of a problem suggested by this drawing. You may choose your outcome, examples like poor fishing stocks, population growth, hunger, or livelihood insecurity might work. You must use some of the conditions noted in the picture, but you could add other issues too. You must include issues at multiple levels – like in the other slide from the nutrition program – showing at least one factor that affects your outcome at the societal, community, household, individual levels. You may use boxes and arrows, circles, whatever makes sense for you. You may want to us a post-it note for each item in your framework, then move the post-it notes around on the large piece of paper until you finish your framework. Finally, copy the framework using the marking pens onto the large paper. One person from the group will be designated to share this framework with the larger group of participants. I will warn groups when there are 10 minutes remaining. Questions? [GIVE 20 MINUTES TO DRAW A MODEL SIMILAR TO THE PREVIOUS SLIDE. IF THEY CAN EXPLAIN THE MODEL – IN WHATEVER FORMAT – AND IT MAKES SENSE, THAT IS GREAT! MONITOR THE GROUPS TO MAKE SURE THEY ARE PROGRESSING AND TO ANSWER QUESTIONS.] [NOTE: IF YOUR GROUP PREFERS TO USE RESULTS CHAINS, ALTER THIS ACTIVITY AND PRESENTATION] ACTIVITY RECOMMENDED MATERIALS FOR THIS ACTIVITY ARE: POST-IT NOTES LARGE PIECE OF POSTER BOARD OR FLIP CHART PAPER; ONE PER GROUP -THICK MARKING PENS; ONE OR TWO PER GROUP [These slides are taken from : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at ]

Brief Presentations & Discussion
Presentation of the conceptual frameworks ~ 5-10 minutes per group Each group presents their framework; Feedback from other participants? As a large group, let’s discuss: What is different about conceptual frameworks for integrated development programs? How can you make the linkages between diverse sectors explicit in your conceptual framework?

Integrated Goal and Objectives
Now that we have our models, let’s move to writing our programmatic goal and objectives showing how we will address the situation.

Philippines: IPOPCORM Program
Community-based Family Planning Community-based Coastal Resource Management [NOTE: YOU COULD USE WHATEVER PROGRAM EXAMPLE YOU WISH HERE – YOU COULD USE AN EXAMPLE FROM THE COUNTRY OR ORGANIZATION WHERE YOU WORK. IF YOU WANT TO USE THIS EXAMPLE, ADDITIONAL INFORMATION IS AVAILABLE. YOU MAY WISH TO ENCOURAGE EVERYONE TO READ A SHORT BACKGROUND ON THE PROJECT: ] Keep in mind the drawing of the situation in coastal Philippines where population, health and environment issues all meet – the one we just used to develop a conceptual framework [SLIDE 13 AND 29]. The IPOPCORM program addresses the situation portrayed in that picture. The program approaches food insecurity with four basic strategies listed on this slide [READ SLIDE]. We will use this programmatic approach as the example to discuss goals and objectives in the coming slides. It is important to note that the activities are multi-sector and reflect an integrated effort to address complex problems. [These slides are taken from : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at ] Policy Advocacy & Community IEC Economic Development (environmentally-friendly) 32

Goals Goal: a broad statement of a desired, long-term outcome of the program. For PHE, it should reflect an integrated outcome. Fundamental to the design of any type of program and its accompanying framework is to clearly identify the program’s goals and objectives. [READ SLIDE] Source: GAP 2003

Integrated Goals: IPOPCORM Example
To improve food security and quality of life in coastal communities while maintaining the biodiversity and productivity of life-sustaining marine ecosystems Writing goal statements can be difficult for PHE/integrated projects because we often try to achieve several broad changes at once. Let’s look at this example from IPOPCORM. [READ SLIDE] Does this seem like an integrated goal? Note that it is not a P, H, or E goal but rather a higher level achievement in food security and quality of life. In PHE, sometimes it is easier to consider writing one overall goal and reflect your integrated approach in the objectives. We will get to this in a bit. For now, let’s talk more about writing a quality goal statement. 34 34

Hints for Writing Quality Goals
Present the desired INTEGRATED outcomes, accomplishments, result or purpose sought (not the process). Capture broad changes in conditions, answering the “so what” question. Often goals reflect behavior, attitude, or economic change and show how our activities contribute toward a larger development impact. Usually goals reflect a result achieved in an intermediate time period (2-5 years). (I want) TO of/among ____________ (action verb) (what) (whom) What behavior/practice, attitude, economic, knowledge, capacity, or access change do you seek? [READ SLIDE]. To write a goal, picture what you want and say: (I want) TO….. [NOTE: HAVE PEOPLE PRACTICE OUT LOUD FOR THE GROUP, COMPLETING THE SENTENCE ON THE OVERHEAD USING THEIR OWN PROJECT OR PROGRAM IDEAS FROM THE CONCEPTUAL FRAMEWORK ACTIVITIES]

Goals Should Be Specific and Tailored!
No!! Yes!! To improve health To improve the health practices of people living in targeted pastoralist communities. To reduce conflict To increase the effective management of critical conflicts among local communities in the Bungoma Biosphere Reserve. To increase standards of living To increase the price of livestock sold on the local market by women in the National Park buffer zone. [READ SLIDE]. Although the “NO!!!” goals are long term, broad changes, the “YES!!!” goals reflect a more tailored idea of the specific changes that you seek.

Objectives Objectives: statements of desired, specific, realistic, and measurable program results. They should be SMART!! Specific: identifies concrete events or actions that will take place. Measurable: quantifies the amount of resources, activity, or change to be expended and achieved. Appropriate: logically relates to the overall problem statement and desired effects of the program. Realistic: Provides a realistic dimension that can be achieved with the available resources and plans for implementation. Time-based: specifies a time within which the objective will be achieved. As we discussed, a goal is a broad statement of a desired, long-term outcome of the program. Objectives are the statements of desired, specific, realistic, and measurable program results. The acronym “SMART” is useful to remember when developing objectives. [READ SLIDE] SPECIFIC: Is the objective specific? Does it cover only one activity rather than multiple activities? MEASURABLE: Is it measurable? Can it be measured or counted in some way? APPROPRIATE: How important is this objective to the work that we are doing? How relevant is it to achieving our goal? REALISTIC: Is the objective actually doable? Can we achieve this goal? TIME-BASED: Does the objective give a timeframe by when the objective will be achieved, or a timeframe during which the activity will occur? Objectives are usually short term – with results in about two years. ATTAINABLE: Sometimes the letter “A” in the SMART acronym stands for attainable. RELEVANT: Sometimes the letter “R” in the SMART acronym stands for relevant: Source: GAP 2003 37

Hints for Writing Quality Objectives
Remember objectives… Are linked directly to the goals and the intended outputs (information, products, processes, services, or results) you hope to produce to reach the goal. Reflect what you hope to produce by undertaking a specific activity. For example: You decide you want to train people (the activity), then the objective will relate to the knowledge you increase. Usually reflect a result achieved in a relatively short time period (0-2 years). Again, writing objectives can be difficult for PHE or integrated projects. Consider that your objectives can address the different sectors, PHE, that your program addresses.

To Write an Objective, Fill In the Blanks:
TO __________ (action/direction of change) the ________ (what knowledge, attitude, practices will be changed) In _______ (targeted population or area of change) by _______ (the expected amount of change in number or percent) as measured by ______(how will you measure the change) by _______ (time frame) Examples To increase the number of children in primary school in Addis Ababa by 25% (as measured by school records) by December 30, 2010. To provide four workshops on peace building methods to 100 community members as measured by field staff workshop reports by December 30, 2008. A properly stated objective is action-oriented, starts with the word “to,” and is followed by an action verb. Objectives address questions of “what”, “who”, “how much”, and “when”, but not “why” or “how.” Are these examples SMART? Are they integrated? [NOTE: ASK PARTICIPANTS TO CRITIQUE THESE OBJECTIVES AS A GROUP. HAVE PARTICIPANTS COME UP WITH AN OBJECTIVE FOLLOWING THIS EXAMPLE BASED ON THEIR OWN PROGRAM OR JUST MADE UP – SHARE AS ONE GROUP]

Integrated Objectives: IPOPCORM Example
Objectives (short term outcomes) By 2004, improve RH outcomes among people living in coastal areas. By 2006, build community capacity to implement coastal resource management including alternative livelihood schemes. By 2007, increase public and policymakers’ awareness and support for integrated PHE approaches. Let’s look at these additional integrated objectives from IPOPCORM as an example. [READ SLIDE]. Are they SMART? If yes, why? If no, let’s make them SMARTer! [FOR EXAMPLE, ASK THEM HOW WILL THESE OBJECTIVES BE MEASURED? DO THEY SEEM REALISTIC IN THE 2-YEAR, SHORT-TERM TIMEFRAME?] [TAKE ~5-10 MINUTES TO COMPLEMENT/CRITIQUE THE OBJECTIVES. WRITE ANY NEW REWRITTEN OBJECTIVES ON THE BOARD, EASEL, OR FLIP CHART.] Are the objectives SMART? Specific, Measurable, Appropriate, Realistic, and Time Limited? 40 40

Goals and Objectives Goal: To improve access to family planning, community health, and natural resource management through integrated development approaches in target communities. Objective 1: Reduce barriers for access to voluntary FP and RH services in target communities. Objective 2: Improve family health in priority biodiversity areas in target communities. Objective 3: Improve community management of natural resources and habitat conservation in target communities. Objective 4: Document and promote sustainable and successful PHE approaches in target communities through improved monitoring and evaluation. [OPTIONAL SLIDE – YOU MIGHT NOT NEED THIS IF YOUR PARTICIPANTS HAVE A GOOD HANDLE ON WRITING GOOD GOALS AND OBJECTIVES. NOTE: THIS EXAMPLE COMES FROM THE GOAL AND OBJECTIVE USED IN THE ILLUSTRATIVE M&E INDICATOR MATRIX INCLUDED AS PART OF THIS TRAINING GUIDE. YOU WILL EXPLORE IT FURTHER IN MODULE 6]. Here is an example of goals and objectives developed as part of a PHE program strategy. Notice that the goal is more general and the objectives are more specific, further explaining the overall goal of the program. Using the SMART acronym, we can assess whether the objectives meet the criteria presented in the previous slides. How could we make these smart objectives? Let’s critique them as a group. [ALLOW 5-10 MINUTES]

Goals and Objectives: Activity
Child malnutrition, death, and disability Outcomes Inadequate Disease dietary intake Immediate causes Insufficient access to food Underlying causes at household/ family level Inadequate maternal & child care practices Poor water/sanitation & inadequate health services [OPTIONAL SLIDE: IF YOU THINK THE PARTICIPANTS NEED MORE GOAL/OBJECTIVE WRITING PRACTICE, TAKE 10 MINUTES AND DO THIS LARGE GROUP ACTIVITY. WITH ALL PARTICIPANTS, DEVELOP A GOAL AND AT LEAST TWO OBJECTIVES FOR A PROGRAM THAT ADDRESSES THIS CONCEPTUAL FRAMEWORK FOR THIS NUTRITION PROGRAM. USE A FLIP CHART OR CHALKBOARD TO WRITE THE SUGGESTIONS. AFTER THE GOAL AND OBJECTIVES HAVE BEEN WRITTEN OUT, REVIEW THEM AGAINST THE SMART CRITERIA: SPECIFIC, MEASURABLE, APPROPRIATE, REALISTIC, TIME-BASED. REFER TO PREVIOUS SLIDES IF NECESSARY. IF THE SMART CRITERIA ARE NOT MET, REVISE GOAL AND OBJECTIVES.] Possible goal and objectives for discussion: Goal: To reduce childhood mortality from malnutrition Objectives: At the end of three years, the proportion of children under five years of age classified as malnourished will be reduced by 25%. At the end of three years, households in target communities will increase food production by 10% At the end of three years, the percent of households planting high quality seed will increase by 10% At the end of three years, the proportion of households with secure access to food will increase by 15%. At the end of three years, the mortality rate among children under five will decrease by 15% (Millennium Development Goals). Quantity & quality of actual resources – human, economic, and organizational – and the way they are controlled Basic causes at societal level Potential resources: environment, technology, people Source: UNICEF, State of the World’s Children, 1998.

Small Group Activity In your groups of 4-6, draw up a simple conceptual framework of your program This will be the basis of your work for the training You will present this framework and talk the group through it (Re) Write your goals and objectives The goal must reflect integration of at least two sectors At least one population/health objective At least one environment/conservation/non-health objective Do your goals and objectives match your model? Are your goals and objectives written effectively? SMART? [ABOUT 1 – 1.5 HOURS. YOU MAY DIVIDE THE GROUP INTO SMALLER GROUPS OR ALLOW THEM TO DIVIDE THEMSELVES. THE GROUPS SHOULD SHARE A COMMON ACTIVITY REGION, PARTNERSHIP, PROGRAM, OR INTEREST AREA IN ORDER TO MAKE THE COURSE MATERIALS AND LEARNING OBJECTIVES APPLICABLE TO THEIR “REAL-WORLD” SITUATIONS. IF AT ALL POSSIBLE, PLACE THEM INTO WORK GROUPS (OR ALLOW THEM TO CHOOSE THEIR OWN) SO THAT THEY ARE WORKING ON THEIR OWN PROJECTS AND CAN LEAVE WITH SIGNIFICANT PROGRESS TOWARDS IMPROVING THEIR ACTUAL PROGRAM M&E ACTIVITIES.] [CONSIDER HAVING THE GROUPS SHARE SOME INFORMATION ABOUT THEIR PROGRAMS AS THE GROUPS WILL WORK ON THEIR PROGRAMS THROUGHOUT THE COURSE. ASK EACH GROUP, AS PART OF THE CONCEPTUAL MODEL EXERCISE TO ALSO PRESENT, BRIEFLY: BRIEF PROBLEM STATEMENT – BULLETED LIST CONCEPTUAL FRAMEWORK/MODEL LONG-TERM GOALS AND RELEVANT OBJECTIVES TYPE OF INTERVENTION – P,H,E, OTHER GEOGRAPHIC SCOPE (I.E. PLACE AND SITE OF THE INTERVENTION) TARGET POPULATION ACTIVITIES AND STRATEGY – FIVE ACTIVITIES LENGTH OF FUNDING IN YEARS [AGAIN, YOU MAY USE A DIFFERENT FORMAT FOR YOUR CONCEPTUAL MODEL. WHATEVER MODEL YOU CHOOSE, MOVE THROUGHOUT THE GROUPS OFFERING HELP WHEN/IF NEEDED.]

Conceptual Framework, Goal, and Objectives Group Report
Each group should present their conceptual framework, goal, and objectives for the group. Walk the group through your conceptual framework, showing the links between problems/issues. Present your goal and objectives. Did you all manage to integrate PH&E components into your conceptual framework, goals, and objectives? [EACH GROUP PRESENTS THEIR MODEL, GOAL, AND OBJECTIVES (AND POSSIBLE PROGRAM DETAILS) – ~10 MINUTES PER GROUP. DISCUSSION ADDS ~20 MINUTES. MAKE SURE EACH GROUP ARTICULATES A HIGHER LEVEL GOAL STATEMENT THAT REFLECTS A PHE APPROACH AND THAT THEIR OBJECTIVES ARE MULTI SECTOR. IF NOT, GENTLY CORRECT THEM IN GROUP DISCUSSION.]

Module 3 Introduction to M&E
[MATERIALS FOR THE MODULE: 1) HANDOUTS OF THE SLIDES, AS NECESSARY; 2) COPIES FOR ALL PARTICIPANTS OF THE MEASURE EVALUATION PUBLICATION, “A GUIDE FOR MONITORING AND EVALUATING POPULATION-HEALTH-ENVIRONMENT PROGRAMS (The PDF of the guide online here: ]; 3) LARGE PIECES OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP, FOR ACTIVITY; 4) THICK MARKING PENS, ONE OR TWO PER GROUP.] Now that we have a greater understanding of how to integrate, why we integrate, how to draw conceptual frameworks of our integrated programs, and how to create strong goals and objectives, let’s move on to monitoring and evaluating our programs as we design, implement, and conclude them. Although this course is not a comprehensive course to teach you everything you need to know about monitoring and evaluation, it is important that we all understand the basics of M&E so that we can make adaptations for the PHE context.

Objectives of this Module
To define monitoring and evaluation To review elements of an M&E plan and necessary steps toward constructing the M&E plan To discuss implementation of M&E plans To discuss specific considerations for M&E in the small-scale, integrated development programming context To introduce “A Guide for Monitoring and Evaluating Population-Health-Environment Programs” Over the next [INSERT TIMEFRAME HERE – 30 MINUTES? 1 HOUR? 2 HOURS?], we will: 46

Monitoring and Evaluation is an Essential Process to:
Determine if the program is on track in carrying out planned activities Decide whether the program is having desired outcomes in all sectors (health, population, environment, gender, livelihoods, etc.) Help make informed decisions about needed corrections, new services, use of program resources, etc. What is monitoring and evaluation? [READ SLIDE]. M&E is an essential process in providing effective and efficient services and ensuring that programs are relevant and successful. For example, it helps us make informed decisions about such questions as appropriate staffing and other necessary resources. M&E helps us know whether a program is being true to its stated goals and objectives. M&E helps us evaluate whether our programs are having their desired impact. If we want to know how a program is performing we might assess it against targets that have been set for specific indicators by the program or funding agency or government. We might assess if a family planning program is reaching its goals in providing counseling to women during postpartum visits and by the percentage of women who are leaders in our conservation efforts. 47

Monitoring is the routine tracking of program activities to help program and project managers measure progress. Are we completing activities according to plan? What are the costs? Who is using our services? Evaluation is a specific study that helps determine program achievement. A well-planned, high-quality evaluation can help answer: How well was the project implemented? Were the desired changes achieved? If the change was achieved, to what extent can it be attributed to the project? What is the difference between monitoring and evaluation? Monitoring and evaluation are related, but distinct, concepts. This slide helps us clarify what question we can answer through monitoring and what types of questions we can answer through evaluation. [READ SLIDE] The important distinction is that monitoring ensures that you are doing what you said you were going to do and helps you make corrections as you implement your program. In contrast, evaluation helps you determine whether you achieved the desired outcomes and, eventually, impact. Note that routine monitoring occurs during the intervention and evaluation is generally used to determine progress towards outcomes. Intervention Outcome 48

Why Do M&E in PHE programs?
To collect information to make informed, evidence-based decisions. To make mid-course adjustments and refine project activities. To demonstrate progress and explain unique PHE-related challenges to stakeholders, funders, & partners. To create records of past and present performance for future assessment and institutional memory. To demonstrate the advantages of working across sectors rather than working in a single sector, showing the value of our integrated approach! Although PHE programs often have few resources and smaller budgets than some traditional single-sector programs, M&E is critical to our success. It helps us to understand the program status in all its complexity. M&E helps us enhance our program performance through evidence-based decision making. It helps us demonstrate accountability for all major functions (service delivery, training, resource mobilization, financing, stewardship) at all levels (from community to national level). How can M&E help you? [READ SLIDE] #1: Are the program’s resources used effectively and efficiently? Is the program on track? Does the program need to change? #2: Where are the successes? Where are we making progress? #3: What happened in the past and what is happening now? #4: You and your field teams collect a lot of data, but how and why should you use it? What can it do for you? How can you convince your field teams and staff to want to collect it? How can we teach our staff that there is something in it for them? #5: There is an urgent need for our field, including decision makers, researchers, and advocates, to demonstrate the value of PHE integration and make a better case for increased investment in this field. What else do you use M&E for? 49

Planning for M&E with an M&E Plan
Links information from various data sources to decisions that will improve programs. States how the program will measure achievements (ensure accountability). Documents consensus (encourage transparency and responsibility). Guides M&E implementation (standardization and coordination). Preserves institutional memory. Measures success. The M&E plan needs to be adjusted when a program is modified. [NOTE: SHOW PARTICIPANTS PAGE 24 IN “A GUIDE FOR MONITORING AND EVALUATION OF POPULATION-HEALTH-ENVIRONMENT PROGRAMS]. What does an M&E plan do? The M&E plan on page 24 describes the strategic information your program will gather and use for decision making that will lead to improved health, conservation, development, etc., programs and ultimately to improved wellness of the community. It is also the fundamental document that will hold the program accountable and tell you whether you succeeded or not. [READ SLIDE] In addition to stating how the program is going to measure what it has achieved, an M&E plan functions to document consensus, guide implementation, and preserve institutional memory. Remember that an M&E plan is a living document and needs to be adjusted when a program is modified or new information is needed. As an additional incentive, know that by completing this training course and the three frameworks, you will be well on your way to completing a draft of your M&E plan! 50

Functions of an M&E Plan: by Programmatic Level
Community: Determine participation levels within all sectors Supervise community-based workers Facility: Facilitate cooperation with local providers/partners Manage logistics among partner organizations District: Request resources/encourage collaboration National: Request resources/advocate for policy changes Your M&E plan can work at many levels depending on your programmatic level and on your partners. At all levels, quality M&E can help you anticipate necessary changes in the program. Within PHE programs, it is imperative that your M&E plan specify how you will manage the monitoring and evaluation of components implemented across sectors and across collaborating partners at all levels. [READ SLIDE]. Within your organizations or programs, how do you plan to collect information from partners at various levels of program or policy implementation? Keep that question in the back of your mind; we’ll return to this issue later in Module 6. 51

When to Plan for M&E The initial project activity plan should include routine monitoring and evaluation. While planning: Identify potential, existing data sources and tools. Create appropriate forms, tools, and guidelines to collect information during implementation. For integrated development programs, plan with partners for shared M&E responsibilities before you begin! M&E should be considered as a critical part of the program design and implementation plan. [READ SLIDE] On page 32 of the “Guide for Monitoring and Evaluating Population-Health-Environment Programs” is a brief overview. We will not be discussing the details, but you may want to read this section for additional information. Looking at the Guide, you can see that the key components are: Define the purpose of the M&E and performance objectives. How much money do you have? Identify indicators, methodological approaches, and data sources. Develop an implementation and data dissemination and use plan. Compile and write the M&E plan. Use data for decision-making and review M&E plan. In this training, we will concentrate on #3, the core of the M&E plan. 52

Considerations for Implementing the M&E Plan in Integrated Programs
Utility Feasibility Propriety Accuracy In implementing the M&E plan think of the following important issues. For PHE, when we often collaborate with partners, it is important that you discuss and consider these issues within your own organization’s M&E plan as well as the M&E plans of your partners: Utility – serve practical information needs of intended users from all sectors. Feasibility – be realistic, prudent, diplomatic, and economical. Think about what your organization can do and how you can share costs with your partners. Propriety – conduct M&E legally, ethically, and with regard to those involved in and affected by the evaluation – are your partners also collecting their data in this manner? Accuracy – reveal and convey technically accurate information – are your partners also collecting data well? 53

Guidance for M&E for PHE
Let’s continue to learn how this guide can provide some useful information and direction for implementers of PHE programs. 54

What’s in the Book? Why Use It?
Overview of PHE, its history, and its rationale Overview of M&E Hints on data quality and collection Steps for developing a M&E plan A set of standard indicators for PH&E sectors, including integration and value added indicators [READ SLIDE] It is important that we define two concepts briefly here and then we will revisit both of these in the coming slides and modules: [FROM THE GUIDE, PAGES 19-20] 1. The definition used here for integration is an indicator that highlights the multi-sectoral nature of your program, demonstrating how you work across the PHE sectors, and/or how you combine messages or interventions. 2. The definition used here for value-added is the identification of indicators that contribute to results across two or more sectors in such a way that outcomes go beyond those anticipated if the interventions had been implemented separately. Indicators that provide information about one or more groups of people who may not have been targeted in the intervention but are still reached through the intervention are considered value-added as well. Value-added indicators are different from integrated indicators, which measure the project’s success in implementing cross-sectoral approaches. If the sectors working together in the PHE strategy are complimentary, their effects may reinforce each other to yield surplus or extra outcomes. Measurements of indicators that could produce value-added information are listed separately in this guide. 55

Guide to Indicators Population Indicators Health Indicators
Environment Indicators Integration Indicators Value-Added Indicators We’ll review this later in great detail in the indicators module, but just so you know about the different ways that we indicate success in our programs, the Guide contains information on indicators from five sectors: 1. Population indicators including those for: Reproductive health; family planning; births 2. Health indicators including those for: Vaccines; safe water; bed nets 3. Environment indicators including those for: Legally-protected area; community-based NRM; species richness 4. Integration indicators including those for: Linked messages; collaboration; awareness 5. Value added indicators including those for: Gender; governance; economic development; youth 56

CHALLENGES FOR M&E FOR PHE

How Do We Demonstrate “Integration?”
Show partnerships and collaborative efforts between sectors Highlight the diverse PHE messages and services provided to communities Demonstrate community buy-in to the integrated approach Promote the sustainability of PHE Indicate the cost-effectiveness of cooperative efforts The Guide is a resource for indicators most frequently used by leading PHE organizations and practitioners in the field. However, although population, health and environment indicators have been used in the field before, one of the challenges of M&E for PHE is how we show and demonstrate our integrated approach and efforts. There are no tried and true measures of integration, in part because we integrate between sectors and link our programs in unique ways. Thus, for M&E of PHE programs, we need to: [READ SLIDE]. The integration indicators in the book [HAVE THEM LOOK AT THE LIST ON PAGE 11], as well as the accompanying discussion about integration, are an attempt to begin to standardize across measurement of these aspects of our programs. It is also a good source for language to highlight the rationale behind integration and what it means to integrate in the PHE context. We will revisit integration indicators throughout the course.

What Do We Mean by “Value Added”?
The value added of integrating PHE, instead of parallel P-H-&E efforts, include our programs’ potential to: Achieve results outside of the initial program plan, including positive effects on, for example, gender, governance, livelihoods, and youth Create broader community-level changes than single sectors programs Provide more bang-for-the-buck for donors than more traditional interventions by saving costs and using funds more effectively [READ SLIDE]. A good example of value-added in a PHE project can be seen in the livelihoods context. Let’s imagine that your PHE project intends to increase the use of condoms and pills by providing contraceptive commodities to local merchants and providing merchants with sales training specific for condoms and pills. Therefore, in the process of reaching the goal of increasing use of condoms or pills, the project may also diversify the livelihood of that merchant by providing an alternative or supplementary stream of income. The diversification of livelihood for that merchant is a value-added result. Also, if the project actively seeks female merchants, or female community-based distributors, the value added of increased condom or pill use could be in the area of gender – promotion of women’s livelihoods, autonomy, and decision-making powers as valued distributors of the methods. [HAVE PARTICIPANTS LOOK AT THE INDICATOR LIST ON PAGE 12.] [HERE YOU COULD LEAD A DISCUSSION ABOUT WHY ADDRESSING PHE SIMULTANEOUSLY IS MORE VALUABLE THAN P-H-E SECTOR EFFORTS FOR YOUR SPECIFIC CONTEXT. ALSO, THIS IS WHERE YOU WOULD HIGHLIGHT HOW YOUR SPECIFIC PHE APPROACH PROVIDES ADDED VALUE FOR DONORS, COMMUNITIES, AND INDIVIDUALS OVER A MORE TRADITIONAL PROGRAM. THE GUIDE MAY BE HELPFUL TO DEVELOP YOUR DISCUSSION, AND MORE DETAIL ON THE VALUE ADDED INDICATORS MAY BE FOUND ON PAGES IN THE GUIDE. ALTHOUGH THESE INDICATORS WILL BE DISCUSSED AGAIN IN THE INDICATORS MODULE, THE TEXT ON PAGES MAY PROVIDE ADDITIONAL POINTS TO CLARIFY THIS FOR THE PARTICIPANTS.]

Special Considerations for M&E for PHE
Multi-sector measurements Limited budgets Need for lessons learned/best practices in M&E Use of both primary and secondary data sources Here are some special issues that we need to consider as organizations and programs interested in integration. We will discuss all of these issues in the coming slides and modules. M&E experience from the field is helpful here. Cara Honzak, of WWF’s Population, Health and Environment program, notes that: “The complexities of conducting M&E across the three sectors of P, H, and E make data collection massive and indicator measurement inherently complex. Each of these fields has a traditional target audience that is distinct. At a minimum: P traditionally involves interviews with women between 15 to 49 years old; H involves interviewing women with children under a certain age (typically 24 months or 5 years); and E traditionally involves interviewing men. Integration expands these audiences even further.” Compounding these issues, we frequently have smaller budgets, limiting our ability to request and solicit technical assistance. Lastly, our creative programs and linked approach is unique, making traditional measures less applicable. The reality is that many of our unique programs have smaller budgets or are considered pilot projects. This consideration forces us to be even more selective in our M&E activities – making sure that our plans are well thought out, effective, and without redundancies to stick to our budgets. Working together, and pooling M&E resources between partners, may help address this challenge. Innovative programs mean that our programs do not have years or decades of example programs with M&E activities that we can base our efforts on. We have to develop pieces of our unique M&E approach as we go. In part, better communication and collaboration between partners and PHE organizations could help us to better understand best practices and lessons learned from the field. Talk to your collaborators, reach out to other organizations, and disseminate your own lessons learned from the field. We are our own best resource! We’ll come back to talking about primary and secondary data briefly in Module 6 when we discuss indicator matrixes and where we get our data. But, overall, PHE programs can not collect all the information and data they need across three or more sectors. Unlike more traditional programs, we rely on our collaborators and partners in the field to help us fill our M&E gaps and complete our M&E plans. Although this type of cooperation and cross-sector work is valuable, it is also challenging. It requires higher levels of communication, transparency, and interdependence. Ensuring quality across partners is more difficult, but not impossible. [ADD ADDITIONAL CONSIDERATIONS FROM YOUR OWN EXPERIENCE IF APPLICABLE] 60

PHE Considerations Continued…
5. Collaboration/cooperation between organizations 6. Demonstrating integration How do we show others how we work across sectors? 7. Issues of “value added” How do we demonstrate that our programs are more than the sum of their parts? Internal relationships between members of one organization are complex – this is compounded by working closely with other people and organizations in a partnership, presenting challenges in managing the politics of different people and organizational styles. It is difficult to “show” how we integrate. As we just discussed, the Guide presents some ways of measuring and demonstrating what “integration” means and how it is implemented. [LOOK AT PAGES FOR MORE INFORMATION ON THESE INDICATORS]. We will address this issue again in the next three modules. Proving the benefits of a multisectoral approach over traditional programs, our added value, is tough, but we need to put effort into demonstrating the advantages of this approach. The key outcomes of the integration are the side effects of this programmatic approach – the way that a holistic approach has positive effects on gender, governance, and youth, for example. Look through the book at the integration and value added sectors for more information on what is meant by these terms. We’ll come back to this in a bit. [LOOK AT PAGES IN THE GUIDE FOR MORE INFORMATION ON THESE CONSIDERATIONS.] 61

Overcoming Challenges of M&E for PHE
The completion of this training and its modules address these challenges: Measurement across sectors Working on a limited budget Alternative assessment techniques Determining the quality of secondary data Measurement of both integration and value added indicators Using standard indicators to measure results [READ SLIDE] This short training course is only the beginning of overcoming these challenges. [USE THESE AS A LIST OF POSSIBLE QUESTIONS TO SPUR DISCUSSION OF OVERCOMING CHALLENGES FOR M&E FOR PHE.] How do you think you can overcome some of these M&E challenges? How and why should you use the data you all regularly collect? What can the information do for you? How can you convince your field teams and staff to want to collect it? How can we teach our staff that there is something in it for them? Considering our proposals, funding, and grants, how can we help our staff understand that improved M&E could lead to increased funding? What are some pro M&E arguments that we can use to motivate our staff?

Special M&E Considerations: Stakeholders
What is a stakeholder? Any person or group with a particular interest in a policy or program In PHE programs, stakeholders may be: Government Program Managers Service Providers Policy Makers Journalists/Media Private Sector Donors Non-Governmental Organizations Beneficiaries Community-based Workers Who else? For M&E to have the desired impact of improving programs, M&E data and information must be used strategically by programs, service delivery organizations, policymakers, and other stakeholders. A stakeholder is anyone who has a “stake” or interest in your program. [READ SLIDE.] We often think of policymakers, funding agencies, and even implementers or providers at the community and facility levels as stakeholders. However, we often do not think of beneficiaries as stakeholders. The people that our programs and services strive to serve make decisions about seeking services and continuing to seek care. It is vital to consider these stakeholders when designing and implementing any program or service. Who are the stakeholders in your program? What might they want to see in your M&E plan? Why might you need to consider your diverse stakeholders as you plan and implement your M&E activities? What other stakeholders can you think of?

Special M&E Considerations: Decision Making
In PHE programs, our complex programs require collaborative decision making in three distinct categories: Advocacy and policy formation (community, district, etc.); Program design and planning; and Program management and operations. [READ SLIDE] What types of decisions fit into categories 1, 2, and 3? How might you need to work with partners to reach consensus on these key issues? Examples: Select priorities for strategic plans, determine advocacy priorities, develop community-based natural resource management plans. Examples: Specification of procedures for wildlife disease management; Procurement of commodities (family planning, bed nets, stoves, etc.); Improving integration of population, health and environment intervention components. Examples: Training and supervision of program staff; Improving performance of family planning promoters; Logistics and supply management; Number of staff to support, program monitoring; Data collection, tracking, resources; Reporting mechanisms; Allocation of regional/district program funds to local implementers. 64

Special M&E Considerations: Data Use
To make decisions well, all decision makers & stakeholders need to understand: How the programs work, including how they collect information. What are the specific reporting and documentation procedures used by all partners? Among partners and staff, who will analyze the data that is collected? Who needs what specific information? How will it be disseminated? Understanding how programs work and the reporting systems of the organizations reporting data can establish effective and useful feedback to key stakeholders at the community, district, and regional levels. It is important to remember that feedback is a two-way, fluid process – data and information moves from the community to the collector to the central office (stakeholders) and back to the community. By understanding your stakeholders’ needs, including the community, your partners and funders, you may be better able to ensure that your staff see the value of collecting/reporting quality data, hopefully ensuring on-time reporting, as well as better quality data. In PHE, as in other programs, we need to engage stakeholders to discuss what information they may find helpful. To get that information to your stakeholders, we can’t forget that training and supervision are needed to improve reporting and documentation, including making sure information moves from collectors to central offices and back to the communities. To do that, we need a common understanding among those who deliver services, those who report on services, those who analyze the data and those who need the information. Only through effective communication can every stakeholder and decision maker have the information they need at the right time to make the right decision. 65

Factors Other Than Evidence-based Information Influence Decisions
Power relationships Timing Competing priorities Public opinion Political ideology Local culture surrounding decision making PHE partners Other information sources Community needs It is important to note that there are many other legitimate factors that influence decision making outside of evidence-based data and information. In the integrated development context, who/what else influences these decisions? 66

Further Readings/Tools on Decision Making, Information Use, and Stakeholder Analysis
Decision Calendar: A tool to help explicitly and systematically align and prioritize decisions with appropriate data resources. Assessment of Data Use Constraints A rapid assessment tool designed to identify barriers and constraints, including those at the organizational, technical, and individual levels. Information Use Mapping Tool A tool to help assess and identify opportunities for improved data use and feedback mechanisms for stakeholders across different levels. Stakeholder Engagement Tool A tool to help ensure that the appropriate stakeholders in decision processes have been identified and involved and identifies key areas of support for scale up. Tools are available at MEASURE Evaluation, search term: DDIU: Stakeholders and decision making is complicated, and it is even more so in PHE programs. Extra resources are available for you. [IF THESE ARE KEY ISSUES FOR YOUR ORGANIZATION OR PARTICIPANTS, CONSIDER USING SOME OF THE RESOURCES MENTIONED IN THIS SLIDE DURING THE COURSE. TRAINING MODULES AND ACTIVITIES ARE AVAILABLE THROUGH THE LINKS.]

ACTIVITY: Starting Group Projects
Return to your groups of 4-6 for the project Discuss the program that you will be working on in the training, including review/creation of: Conceptual framework, goals, objectives Again, if you don’t have an existing program, you may design an example program Write (or develop) an abridged program plan/description to use as a basis for your M&E work Include at least 10 key activities to reach your goal/objectives (must include P,H, & E activities) Project timeline – how long will your program be funded? Anticipated partners Key stakeholders/decision makers [~45 MINUTES, LESS IF THEY ARE WORKING ON AN EXISTING PROGRAM. THEY SHOULD RETURN TO THE SAME SMALL GROUPS AS FOR THE CONCEPTUAL MODEL EXERCISE.]

Frameworks, Part II Logic Models
Module 4 Frameworks, Part II Logic Models [MATERIALS FOR THE MODULE: 1) HANDOUTS OF THE SLIDES, AS NECESSARY; 2) COPIES FOR ALL PARTICIPANTS OF THE MEASURE EVALUATION PUBLICATION, “A GUIDE FOR MONITORING AND EVALUATING POPULATION-HEALTH-ENVIRONMENT PROGRAMS (The PDF of the guide is online here: 3) PRINTOUTS OF THE POWERPOINT SLIDE “BLANK LOGIC MODEL” FROM THE TRAINING ACTIVITY FOLDER; 4) LARGE PIECES OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP, FOR ACTIVITY; 5) THICK MARKING PENS, ONE OR TWO PER GROUP.]

Type of Framework Brief Description Use for Program Management Basis for Monitoring and Evaluation of PHE Conceptual Shows the interactions of various factors that operate in your program environment Determine which factors the program will influence and how Helps explain and promote the multi-sector influences and integrated approach Logic model Logically links inputs, processes, outputs, and outcomes of planned activities Shows the causal relationship between inputs and reaching the objectives Shows intended progress of activities at all stages of the program – aids indicator selection Indicator matrix Logically links program goal, objectives, outputs, outcomes, and activities Shows the causal relationship and steps between activities and meeting objectives Guides monitoring and aids assessment by stating the information you will collect, how, when, where, and why In Module 2, we talked about conceptual frameworks, helping you lay out the problem statement, identify causes within multiple sectors (PH&E), and show your rationale for an integrated approach to solve part of the problem. In this module, we will create logic models to help show the logical connections between the: inputs (what you need before you start your program) processes (activities – what your program will do) outputs of an activity (what is the immediate result of your program activity) and its link to the program objectives (outcomes – what changes in knowledge, attitudes, and practices you might see over 2-3 years) and goals (long term impact) [READ LOGIC MODELS SECTION ON SLIDE]

Logic Models Definition:
Diagrams that identify and illustrate the linear relationships flowing from program inputs, processes, outputs, and outcomes. Inputs or resources affect processes or activities which produce immediate results or outputs, ultimately leading to longer term or broader results, or outcomes. Purpose: Provides a streamlined interpretation of planned use of resources and desired ends Clarifies project/program assumptions about linear relationships between key factors relevant to desired ends Other terms used: M&E Frameworks, indicator matrixes Logic models are diagrams that identify and illustrate the linear relationships flowing from program inputs, through processes, outputs, and leading to outcomes. Inputs, or the program’s resources, provide the fuel for Processes, or the program’s activities. These processes produce immediate or direct Outputs or Effects; in other words, the program’s activities yield some intermediate results. These Outputs lead to long-term or broader, overall results, or the program’s Outcomes. A logic model is a useful tool for identifying and illustrating operational factors important to achieving a successful ultimate outcome. For this reason, they are sometimes perceived as the most useful for program monitoring. Since it is the only framework that specifies inputs, it is also the only model that can directly point toward indicators for process monitoring. One of its strengths is that it can help identify linkages and key processes, highlighting the relationship between resource allocation and success, as well as where problems may lie if goals are not being achieved. Logic models provide more detail about the activities and the necessary inputs than conceptual frameworks do. Logic models are also sometimes called by other names. Although terminology regarding frameworks may vary from organization to organization, what is important is to understand the components of this model and how to think through it, whatever name you wish to use. [BE CONSISTENT!] [“LOGIC MODELS” ARE USED IN MANY ORGANIZATIONS, AND PARTICIPANTS MAY HAVE SPECIFIC NOTIONS ABOUT THE LABELS THAT SHOULD BE AFFIXED AT EACH LEVEL (E.G. OUTPUTS VERSUS EFFECTS). IT IS UNLIKELY TO BE A PRODUCTIVE USE OF TIME TO DEBATE THESE SEMANTICS FROM THE VARIOUS ORGANIZATIONAL PERSPECTIVES. THE POINT HERE IS TO EXPLAIN AND ILLUSTRATE THE GENERAL IDEA AND USEFULNESS OF LOGIC MODELS SO THAT PARTICIPANTS SEE THAT THEY CAN HELP CLARIFY THE LINEAR FLOW CONNECTING RESOURCES, ALLOCATION DECISIONS, AND PROGRAM EFFECTIVENESS.]

Key Terms: Inputs: human, financial, social, political, or other resources needed to begin and complete the program. Processes: activities which produce immediate results – what will you be doing? Outputs: immediate results from your processes – what are the results of your activities? Outcomes: longer term or intermediate results – the progress towards achieving your objectives. Impact: long-term changes necessary to sustain your goal. Although you may use other definitions, for the purposes of this course and MEASURE Evaluation, these are the definitions of these key terms. [READ SLIDE]. Generally, in M&E within our integrated development programs, we are concerned most with measuring outputs and outcomes. Impact is much harder and expensive to measure and often too long term for the funding cycles in which we operate.

Logic Model for Integrated Development
For example, here is a generic logic model for a PHE program – it represents many possible ways that you could fill in under these headings. [READ THROUGH SLIDE COMPONENTS AND ASK FOR COMMENTS/QUESTIONS – YOU MAY CREATE YOUR OWN LOGIC MODEL INSTEAD.]

Logic Model: Training Activity
INPUT PROCESS OUTPUT OUTCOME IMPACT Times and skills to develop new agricultural technology training curriculum Conduct training events Local farmers trained in new techniques Increase in agricultural production from those who adopt new technologies Increased food security in target population Let’s start with one activity as an example, and let’s talk through the arrows as part of the process of creating the model. This logic model presents a straightforward view of a project component designed to reduce food insecurity by training farmers in a new agricultural technology. As you can see, it does not try to account for all of the factors that may be influencing operation and results like a conceptual framework, but instead focuses on the project’s immediate activities and effects of those activities. This narrow focus assists program managers and M&E planners as they clarify the direct relationships among elements of particular interest within a particular program effort. In this example, using arrows, you can talk through how you think your input will influence your process, which will affect your output, which will lead to your outcome and eventual impact. It operates in an “if – then” format. For example, we could say: “If we put in the time and skills to develop a new agricultural technology training curriculum, then, we could conduct training events with the curriculum, the process, or activity. If we conduct those processes, activities, then the local farmers will be trained in the new technique – the output, or immediate result of the activity. If they are trained, then, they will be able to increase their agricultural production using those new technologies – the short-term change in practice as a result of our program. If they increase production in the short term, then, in the long term (impact), they will increase food security.” Make sense? Any questions?

Logic Model: Family Planning Activity
INPUT Human and financial resources Demand for FP in the community PROCESS Educate men and women about the advantages of modern method use Distribute FP methods in the community Train program staff in providing FP information and methods OUTPUT Sessions held in community about family planning methods Increased interest in FP FP methods distributed in communities Clinic staff trained in FP method counseling OUTCOME 1. Increased access to contraceptive methods 2. Increased access to FP counseling 3. Increased number of new users of modern methods 4. Increased male participation in FP decisions IMPACT Increased contraceptive prevalence This logic model shows a more complicated set of activities within one sector. Each major activity, intermediate result, or objective should have its own logic model. In this example, the input is the human and financial resources to increase family planning use. The process, or program activities related to this will be the distribution of methods, educating women, and training program staff. The anticipated outputs are that the educational meetings are held, increased interest, training held, methods distributed, etc. The outcomes are increased access to FP, increased access to counseling, etc. All this would lead to increased contraceptive prevalence in the community. Again, we could add arrows to connect specific inputs, processes, outputs, outcomes, and impact. [ASK FOR A VOLUNTEER TO PROVIDE AN EXAMPLE OF WHERE YOU COULD PLACE ARROWS BETWEEN COMPONENTS TO CONNECT ALL THE PIECES AS IN THE PREVIOUS EXAMPLE. THERE ARE MANY WAYS TO TALK THROUGH AN EXAMPLE. ONE EXAMPLE COULD BE: IF THERE IS DEMAND FOR FP IN THE COMMUNITY, THEN WE COULD EDUCATE WOMEN ABOUT THE ADVANTAGES OF MODERN METHODS – OUR PROCESS/ACTIVITY. IF WE EDUCATE WOMEN, THEN WE WOULD HOPE TO INCREASE INTEREST IN USING THE FP METHODS. IF WE INCREASE INTEREST, THEN IT IS HOPED THAT WE WOULD INCREASE THE NUMBER OF NEW USERS OF MODERN CONTRACEPTIVES. IF WE INCREASE THE NUMBER OF NEW USERS, THEN, WE WOULD ULTIMATELY INCREASE CONTRACEPTIVE PREVALENCE.]

Logic Model: Diarrhea Prevention Activity
INPUT Human and financial resources Demand for disease intervention in community PROCESS Train park guards in diarrheal disease and use of oral rehydration salts Train local community members in safe water storage Form natural resource management committees in 10 communities near park OUTPUT (ORS) packets distributed Community members trained in safe water storage NRM committees formed OUTCOME 1. Reduction in acute diarrhea cases among children 2. Increased safe water storage 3. Involvement of NRM committees in promotion of safe water IMPACT Reduced morbidity /mortality from diarrheal disease among children Here is another example of a logic model. This one represents a diarrhea prevention component for an integrated program aimed at reducing childhood disease due to unsafe water. Imagine this project takes place in communities with high diarrhea incidence and that the community is located within a national park buffer zone. This component is both Health and Environment because the community is located in a national park and park guards are involved in outreach activities. The inputs, processes, outputs, outcomes, and impacts are only a piece of a larger puzzle and are not intended to be perfect or complete. [READ THROUGH COMPONENTS ON SLIDES. AGAIN, ASK FOR A VOLUNTEER TO “CONNECT THE DOTS” USING “IF-THEN” BETWEEN BOXES BY WALKING THROUGH AN EXAMPLE OF THE LINKAGES AS WE DID ON THE PREVIOUS SLIDE. MAKE SURE EVERYONE UNDERSTANDS HOW TO LINK THE BOXES.] 76

Logic Model: Voluntary Counseling and Testing for HIV
INPUTS PROCESSES OUTPUTS OUTCOMES IMPACTS Financial resources Supervision and training for VCT personnel People know their HIV status Clients (HIV+ and -) develop & adhere to personalized HIV risk-reduction Risk behaviors decreased HIV transmission rates decreased Demand for services Provide pre-test counseling HIV+ people are referred to appropriate services Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples HIV+ clients develop & adhere to personalized HIV care, support, and treatment plans Infrastructure HIV incidence decreased Provide HIV testing VCT MIS Provide post-test counseling Client records are available VCT protocols, guidelines, and training documents Health outcomes of HIV+ improve HIV morbidity & mortality decreased Refer HIV+ clients to appropriate services (PMTCT, care and support, treatment) Service reports are produced Continuity of care is available Counseling and testing personnel Maintain patient records Program and services are improved [THIS IS AN OPTIONAL SLIDE TO SHOW THE COMPLEXITY OF LOGIC MODELS, USING AN HIV INTERVENTION AS AN EXAMPLE. THIS EXAMPLE SHOWS HOW, WITHIN A PROGRAM, THE OUTPUT OF ONE PROGRAM ACTIVITY COULD BE AN INPUT FOR ANOTHER ACTIVITY. THE MODEL ALSO SHOWS HOW MUCH LARGER AND MORE COMPREHENSIVE THE LOGIC MODELS BECOME AS YOU ADD ACTIVITIES AND CONSIDER LARGER SECTIONS OF YOUR OVERALL PROGRAM. YOU MAY CHANGE THIS EXAMPLE FOR A MORE COMPLEX LOGIC MODEL THAT MEETS YOUR NEEDS OR YOU MAY DELETE THIS SLIDE.] This is an example of a logic model for a VCT (voluntary counseling and testing for HIV) program. The inputs, processes, and outputs of VCT service delivery are presented in a logically connected manner. You could add arrows to show connections. For example, you might draw an arrow from the input if we develop “VCT protocols, guidelines, and training documents” to the process of then we could “provide pre-test counseling” to the outputs then “people know their HIV status” to the outcomes then we could help “Clients (HIV+ and -) develop & adhere to personalized HIV risk-reduction.” Note that as you draw the arrows, you will fill in the blanks through statements like, “if – then” just like the examples from the previous slides. Overall, these models are intended to help you plan and logically think through your programs from inputs to outcomes. Programs might develop logic models for each activity or each sector. You might display these components like this example, or you might have a different format for showing these linkages. It is completely up to you. HIV test kits Complete reporting requirements Referral system for prevention & Tx services

Group Activity: Creating a Logic Model PHE Project in Nepal
Goal: Improve the quality of life of people living in forest corridors and ecosystems while insuring sustainable management of Nepal’s natural resources Objective 1: Health outcomes of forest-dependent communities improved by 2008 Objective 2: Enhance capacity of user groups to promote human and ecosystem health by 2007 Objective 3: Increase support In forest user groups for PHE by 2008 Activity 1: Promote Family Planning as a Sustainable Livelihood Approach (SLA) using Traditional Media (street Drama) Activity 2: Develop Community- based Distribution System for FP Products Activity 3: Disseminate FP HIV/AIDS & PHE Information through Non-formal Education Channels Activity 4: Promote the Use of Improved Cooking Stoves Create Livelihood Opportunities that are Eco-friendly (i.e., briquette making from invasive plant species) Activity 5: Empower Community Forest User Groups (CFUGs) to Manage Forest Assets and Integrate P&H Interventions into Their Five-year Operational Plans [NOTE: YOU MAY SKIP THIS AND MOVE ON TO THE NEXT ACTIVITY IF YOU THINK THAT YOUR GROUP UNDERSTANDS LOGIC MODELS WELL.] This is a simple programmatic framework from a PHE project in Nepal. The format is not important. In this example, notice how the activities are linked to accomplishing the objectives which will aid attainment of the goal. Activity: 1. Using this slide, break into partners (groups of two persons). 2. In each pair, create a simple logic model (like the training example). 3. Take one activity from the five examples in this slide. [NUMBER OFF SO THAT ALL FIVE ACTIVITIES ARE ADDRESSED] 4. Using the one activity from this slide, create a simple logic model including one input, process, output, outcome, and impact. 5. Note: the activity (process!) is provided. 6. Also note: the outcome should be very similar to the objective! 7. Therefore, you just need to come up with the input, output, and impact and put it in the logic model format 8. Remember the “if-then” format. [GIVE ~10-15 MINUTES FOR WORK, AND GIVE ~15 MINUTES FOR PRESENTATION/DISCUSSION. ASK THEM TO USE “IF-THEN” STATEMENTS.] [This slide is taken from : Session 2: Commonalities, Conditions & Characteristics of PHE Projects. More information available at

Small Group Activity Return to small groups
Determine/select three activities for your program: One activity from P or H One activity from E One activity that may work as integration or value added. Give it a try! Use the Guide for help. Develop logic model components for the three activities/processes of your program Work through inputs, processes (activities), outputs, outcomes, and impact After ~30 minutes, a member of each group will share one logic model Return to your small groups for these framework activities. Now you will practice developing your logic models. Select three activities from your project. Then, using a series of “if-then” statements, create a logic model for each activity with at least one input, process, output, outcome, and impact. You will have ~30 minutes for this activity. Then, each group will share one logic model with the larger group. I will let you know when 10 minutes remain. At the end of this activity, you should have one logic model written out on the large paper to be put on the wall. As you are completing this activity, be sure to refer back to your conceptual framework. [MONITOR THE GROUPS TO MAKE SURE THEY ARE PROGRESSING AND TO ANSWER QUESTIONS.] [RECOMMENDED MATERIALS FOR THIS ACTIVITY ARE: COPIES OF THE POWERPOINT SLIDE “BLANK LOGIC MODEL” FOR EACH INDIVIDUAL OR GROUP LARGE PIECE OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP THICK MARKING PENS, EACH GROUP SHOULD HAVE PENS FROM THE PREVIOUS ACTIVITY.]

Module 5: Indicators [MATERIALS FOR THE MODULE: 1) HANDOUTS OF THE SLIDES, AS NECESSARY; 2) COPIES FOR ALL PARTICIPANTS OF THE MEASURE EVALUATION PUBLICATION, “A GUIDE FOR MONITORING AND EVALUATING POPULATION-HEALTH-ENVIRONMENT PROGRAMS (The PDF of the guide is online here: 3) PRINTOUTS OF THE POWERPOINT SLIDE “BLANK LOGIC MODEL_WITH INDICATORS” FROM THE TRAINING ACTIVITY FOLDER; 4) PRINTOUTS OF THE WORD DOCUMENT “INDICATOR REFERENCE SHEET_GUIDED” FROM THE TRAINING ACTIVITY FOLDER; 5) LARGE PIECES OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP, FOR ACTIVITY; 6) THICK MARKING PENS, ONE OR TWO PER GROUP.]

Indicators Module Objectives
At the end of the session, participants will be able to: Critique indicators Identify criteria for selection of sound indicators Understand how indicators are linked to frameworks Select indicators and complete an Indicator Reference Sheet This is a critical module for M&E. At the end of the session, participants should be able to critique indicators, identify criteria for selection of sound indicators, understand how indicators are linked to the frameworks covered in the previous module, and select indicators and complete an Indicator Reference Sheet.

What Is an Indicator? An Indicator is a variable that measures one aspect of a program/project or outcome. An appropriate set of indicators includes at least one indicator for each significant aspect of the program or project. ***For PHE, try to always include integration and value-added indicators too*** An indicator is a variable that measures one aspect of a program or project. Let’s take a moment to go over each piece of this definition. The purpose of indicators typically is to show that program activities are carried out as planned or that a program activity has caused a change or difference in something else. Therefore, an indicator of that change will be something that we reasonably expect to vary. Its value will change from a given or baseline level at the time the program begins, to another value after the program and its activities have had time to make their impact felt, when the variable, or indicator, is calculated again. Secondly, an indicator is a measurement. It measures the value of the change in meaningful units for program management: a measurement that can be compared to past and future units and values. A metric is the calculation or formula that the indicator is based on. Calculation of the metric establishes the indicator’s objective value at a point in time. Even if the factor itself is subjective, like attitudes of a target population, the indicator metric calculates its value objectively at a given time. Thirdly, an indicator focuses on a single aspect of a program or project. It may be an input, an output, or an overarching objective, but its related metric will be narrowly defined in a way that captures that aspect as precisely as possible. A full, complete, and appropriate set of indicators for a given project or program in a given context with given goals and objectives will include at least one indicator for each significant aspect of program activities. For PHE, it is important that we attempt to prove the organizational efficiencies inherent in a multi-sectoral or partnering approach to development. These types of measures help us indicate the project’s success in implementing an approach that yields outcomes beyond those anticipated for single-sector population, health, or environmental programs. The integration and value added indicators can help us show that our programs have more “bang for the buck,” both in terms of cost effectiveness and broader, more positive impacts. They demonstrate our unique contribution and should be used as much as possible.

Characteristics of Good Indicators
Valid: accurate measure of a behavior, practice, or task Reliable: consistently measurable in the same way by different observers Precise: operationally defined in clear terms Measurable: quantifiable using available tools and methods Timely: provides a measurement at time intervals relevant and appropriate in terms of program goals and activities Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact What makes a good indicator? Fundamentally, good indicators must be valid and reliable measures of the result. The other desirable characteristics listed here all serve in a sense as aids that help guide the design of indicators and metrics toward this ideal or goal of valid, reliable indicators. Valid: An indicator is valid when it dictates an accurate measurement of the activity, output, or outcome of the program. Reliable: An indicator is reliable when it minimizes measurement error, that is when it is possible to measure it consistently over time, regardless of the observer or respondent. Precise: Indicators should be operationalized with clear, well-specified definitions. Timely: Indicators should be measured at appropriate intervals relevant in terms of program goals and activities. Comparable: Where possible, indicators should be structured using comparable units, denominators, and in other ways that will enable increased understanding of impact or effectiveness across different population groups or program approaches. For PHE, using the Guide helps make our M&E efforts comparable across programs or projects. Next, each of the characteristics of good indicators will be discussed and examples of good and problematic indicators identified. [THESE TECHNICAL ISSUES WILL BE COVERED AS IDEALS TOWARD WHICH TO STRIVE. LATER SLIDES WILL DISCUSS A NUMBER OF CAVEATS AND TRADE-OFFS OFTEN NECESSARY IN TERMS OF THE PRAGMATIC MANAGEMENT OF PROGRAMMATIC M&E.]

Characteristics of Good Indicators: Valid
Indicator measures what it is supposed to measure Accurate measure of a behavior, practice, or task Direct measures Proxy measures (substitute measures) Straightforward interpretation: change in value signals a change in concept or behavior Validity may be the most important characteristic of a good indicator. A valid indicator is one that accurately measures the phenomenon (the task, behavior, or practice) that it is designed to capture. The indicator provides direct and focused information about the target or result it aims to measure. The validity of an indicator is an attribute that can only be assessed in the context of considering the result or phenomenon it is aiming to measure. Sometimes valid indicators that could be selected or designed for particular important results may be impossible to use for practical reasons such as costs or other material or logistical obstacles that may prevent collecting all of the data that would be necessary. In that case the next best thing is a proxy indicator. A proxy indicator is one that does not capture the exact concept or single aspect of your activity’s result, but aims to measure a concept that approximates the true or ideal indicator. Your M&E plan should make sure to note where you will be using proxy indicators and the reasons. It may become possible later to collect other data and construct a valid indicator for that result, in its place. On the other hand, if uncertainty exists about data collection for some results, it may be prudent to think of proxy indicators for which the data may be easier or cheaper to collect. A good example of a proxy measure may be in the area of family planning. Although contraceptive prevalence is a good indicator, it requires population-based information. A good proxy for your program could be # of new users to modern contraceptive measures. It gets at the same idea but reflects our limited budgets. Another example may be that if you want to indicate success in a park expansion project, you may want to collect area of legally protected habitat. But, that requires more time and more effort. Instead, you measure your program using the indicator of “# of park infractions reported” showing an increase in governance. Not an exact measure, but a substitute in the general “spirit” of the indicator.

Validity: Class Discussion
Is “contraceptive prevalence rate” a valid indicator of the impact of a family planning program for small scale programs? Is “number of trees planted” a valid measure of reforestation efforts? Is “# of women on natural resource management committee” a valid measure of women’s participation? Does “new users of modern contraception” measure women’s attitudes towards family planning? Let’s discuss the following examples as a group. Keep in mind what you are trying to measure versus what you are actually measuring! [HINTS FOR DISCUSSION ANSWERS:] NO. CONTRACEPTIVE PREVALENCE RATE RELIES ON POPULATION-BASED SURVEYS AND HIGHLIGHTS CONTRACEPTIVE USE FOR LARGE GROUPS OF WOMEN. IN CONTRAST, FOR A SMALLER PROGRAM, YOU MAY NEED TO SHOW IMPROVEMENTS ON A SMALLER SCALE – SUCH AS INCREASES IN INTEREST, FIRST TIME USERS, OR MEN’S INVOLVEMENT – INDICATORS THAT RESPOND BETTER OVER A SHORT PERIOD OF TIME. NO. HOW DOES JUST PLANTING TREES LEAD TO REFORESTATION? DID YOU PLANT 1,000 TREES IN 1,000 ACRES? IS THAT REFORESTATION? DID THE TREES SURVIVE? WERE THE TREES APPROPRIATE FOR THE CLIMATE OR AREA? DO THE WOMEN SIT ON THE COMMITTEE OR DO THEY ACTUALLY PARTICIPATE IN THE DECISION MAKING? IF THEY ARE ON THE COMMITTEE BUT HAVE NO VOICE OR THEIR VIEWS ARE DISREGARDED, IS THAT “PARTICIPATION?” HOW DO YOU KNOW WOMEN’S ATTITUDES JUST FROM KNOWING THE NUMBER OF NEW USERS?

Characteristics of Good Indicators: Reliable
Consistently measurable in the same way by different observers or people Types of errors in measuring indicators: Bias: estimates of condom use may reflect people’s desire to answer questions to please you! Personal opinion: indicators that ask for personal judgment such as “quality,” “positive environment,” and “progress.” Reliability, or minimization of measurement error, is another important characteristic of an indicator. Measurement error is just as it sounds – errors or inconsistencies in measuring an indicator. These types of errors are critical because indicators are used to assess program performance. If changes in indicator values merely reflect errors in their measurement, conclusions about program efficiency or effectiveness will not be accurate. All indicators and metrics need to be examined critically to assess ways to reduce measurement error that might occur. In population, health and environment programs, problems in measurement may commonly arise from subjectivity, or asking people opinions or participation. Subjectivity introduces measurement error because the indicator’s value will be influenced by the impressions and sentiments of the measurer – values will not be comparable over time or across geographical units or populations. On the slide are a few examples of measurement error. Many “quality” indicators (quality of care, leadership, supervision, etc.) call on the personal judgment of the data collector or analyst to determine what “quality” is, and this could vary by individual. Another example is policy “environment” and other political “progress” indicators – who defines “environment” and “progress.” These terms might mean different things for different people, and opinion might vary. Try to avoid indicators that solicit someone’s opinion or define what each word means (quality, progress, management, etc.) for increased accuracy. [IF DESIRED, TECHNICAL DETAILS OF MEASUREMENT ERROR COULD BE COVERED IN DEPTH IN ADDITIONAL MODULES ON, FOR EXAMPLE, SURVEY TECHNIQUES, OR IMPACT ASSESSMENT. OTHER ISSUES LIKE SAMPLING ERROR AND NON-SAMPLING ERROR ARE TOO COMPLEX FOR THIS TRAINING.]

Characteristics of Good Indicators: Precise
Operationally defined in clear and understandable terms Partner Activity: each two-person group will develop two definitions: Select from: New user to modern contraceptives Knowledge of PHE issue Adoption of improved agricultural technique Participated in income generation activity An indicator should be defined in precise, unambiguous terms that clearly describes exactly what is being measured. Where practical, the indicator should give a relatively good idea of the data required and the population among which the indicator is measured. This seems like an obviously desirable attribute of indicators, but deserves emphasis since many indicators in common use are not well-defined in clear terms, or at least include terminology that could be improved to add greater precision. For instance, “new user”, “knowledge of new technology”, “adoption of new technology”, or “trained provider” can all mean and imply different things in different circumstances. The more you can spell out in the indicator, the less room there will be for later confusion or complications. Always specify the details! [INCREASING THE CLARITY OF THE WORDING OF INDICATORS CONTRIBUTES TO VALIDITY IN THE USE OF THE INDICATOR. THE MORE CLEARLY YOU SPECIFY THE OPERATIONAL ASPECT OF THE RESULT THAT YOU INTEND TO MEASURE WITH THIS INDICATOR, THE SIMPLER IT BECOMES TO ENSURE THAT CONCEPTS MATCH MEASUREMENT.] [ACTIVITY: PARTICIPANTS INDIVIDUALLY DEFINE THE TERMS LISTED ABOVE, OR OTHERS, AND WRITE THESE DEFINITIONS DOWN ON SCRATCH PAPER. THEN, DISCUSS WITH THE ENTIRE GROUP TO COMPARE THEIR DEFINITIONS. THE DIFFERENT DEFINITIONS PARTICIPANTS DEVELOPED WILL SHOW HOW MANY DIFFERENT IDEAS CAN BE CONVEYED BY TERMS THAT ARE OFTEN CONSIDERED ‘OBVIOUS’ WHEN USED IN INDICATORS. THIS SHOULD HIGHLIGHT THE IMPORTANCE OF A PRECISE DEFINITION OF AN INDICATOR.] [POSSIBLE ANSWERS/DISCUSSION POINTS ABOUT THE DEFINITIONS FOR GROUP DISCUSSION: IS “NEW” NEVER USED OR NOT USED WITHIN THE LAST 1-5 YEARS? WHAT DOES MODERN MEAN? YOU MUST SPECIFY THE DETAILS! HOW WILL YOU MEASURE KNOWLEDGE? DO THEY TELL YOU ABOUT IT? DO YOU ASK MEN OR WOMEN TO NAME A PHE ISSUE SPONTANEOUSLY OR DO YOU ASK PEOPLE TO LINK ISSUES? ACCURATE KNOWLEDGE? WHAT DOES “ADOPT” MEAN? TRIED IT ONCE? USE IT FREQUENTLY FOR A CERTAIN NUMBER OF MONTHS? DOES IT MEAN CORRECT ADOPTION OR COMPLETE ADOPTION? WHAT DOES “PARTICIPATE” MEAN? ATTENDED A SESSION? ADOPTED A NEW FORM OF INCOME GENERATION? MADE MONEY?]

Characteristics of Good Indicators: Measurable
Quantifiable using available tools and methods Examples of measures: Days of drug stock outs Number of hectares under improved management % of children presenting at health clinic with acute diarrhea % of households participating in income generating activities Since the intention is to calculate the indicators that are defined as part of an M&E plan, it is important to make sure that it is possible to measure these indicators in a quantifiable way. Important questions to answer, in addition to those related to financial and human resources for obtaining the data, are: Are tools and methods available to collect or calculate this information? If not, can they be developed for this project or program? Do you have the information you need for both the numerator and the denominator? Some examples of indicators that cannot be measured easily or with little cost are: – # of days of a drug stock out: facilities may not collect information on the number of days a drug was not available. Sometimes facilities record this information as a yes/no variable, indicating whether or not there was any stock out in each quarter. – # of hectares under improved management. You would need a very precise definition of “improved management,” not to mention that you could need to know the exact calculation of how many hectares are “improved.” If this is at the community level, what will you do with the ½ hectare? Will you report by household, district, community? – Who records the information at the health clinic? Do they report the difference between acute diarrhea from cholera and acute diarrhea from malnutrition? Does “presenting” mean “treating”? Were they treated properly? Do you know how many households are in your community? Is it any income generating activity or the one from your specific program? What do you mean by participation? If both the man and the woman participate from one household will that be counted the same as another household where only the man participates? Think about whether you can measure these indicators and whether they seem like good options for you. There is no right or wrong answer; every program is unique.

Characteristics of Good Indicators: Timely
Provides a measurement over period of time with data available for all appropriate intervals Timeliness Considerations: Reporting schedules Recall periods Survey schedules Length of time over which change can be detected Timeliness of an indicator is crucial, as it affects not only the indicator itself but the data collection schedule and the reporting schedule as well. Indicators that are tools for measuring results require data that should be collected after a period sufficient for program activities to have made a measurable impact. Although this may seem obvious or self-evident, many times data collection may be affected by the reporting schedules of others: the government’s, other partners’, and or headquarters'. So to the extent possible, these logistical factors should be taken into account in indicator design. For instance, if your condom-social-marketing partners compile routine statistics every three months, it is better not to design an indicator counting condoms distributed in the last 30 days! That data would not exist! Similarly, if your conservation project aims at improving reforestation efforts, you cannot measure forest regrowth in a two-year project cycle. You would need a faster-responding indicator to show your process/outputs, such as trees planted or hectares protected or another indicator that could be collected in a timely fashion. An advantage of integrated projects is the way that population or health indicators may show faster results, making these partnerships advantageous for conservation organizations with timing issues in reporting progress over short terms. Other timeliness factors to take into account would include the periods for which reporting sub-units, such as clinics, may compile statistics; the recall period which surveys should rely on respondents’ memories or retrospective evaluations; the periodicity of existing survey exercises; and the length of time one could reasonably expect change in some variables, such as mortality or fertility rates, over a country’s population. [TIMELINESS CONTRIBUTES TO RELIABILITY. IF YOU ARE TRYING TO MEASURE IMPACTS THAT HAVE NOT HAD TIME TO OCCUR OR HAVE OCCURRED OVER SUCH A LONG PERIOD OF TIME THAT MANY OTHER FACTORS WILL HAVE INTRUDED, TAKING TIMELINESS INTO ACCOUNT IN AN M&E PLAN SHOULD INVOLVE COMMUNICATION WITH THE FULL RANGE OF STAKEHOLDERS IN THE M&E PROCESS REGARDING LOGISTICS, REPORTING, AND THEIR M&E NEEDS. ENGAGING DIVERSE PARTNERS IN THE M&E PLANNING PROCESS.]

Characteristics of Good Indicators: Programmatically Important
Linked to a programmatic impact or to achieving the objectives needed for impact 1. Example: Program to distribute and promote condoms Indicator: # of sex partners in last four weeks 2. Example: Fuel efficient stove distribution program Indicator: % of community households with fuel efficient stoves 3. Example: Program to increase access to oral rehydration salt (ORS) through community based distributors Indicator: # of ORS packets distributed by clinics in past month Since the purpose of an M&E plan and the indicators developed for it is to monitor a program, indicators should be linked to the program activities. If the indicator is related to factors the program is not attempting to affect, no useful information for the program will result from collecting and reporting this indicator. To be most useful, indicators should be designed to inform specific decisions that need to be made within a program in order to make it most useful. Discussion [ALL INDIVIDUALS TO THINK ABOUT THE INDICATORS FOR 1-2 MINUTES BEFORE ASKING FOR A RESPONSE. THEN, USE THESE SUGGESTIONS FOR THE DISCUSSION:] On the slide, the first example is of an indicator that does not pertain to the program activities or desired outcomes. In a program consisting only of condom promotion and distribution, an indicator to measure number of sex partners in past four months would not provide information that would help monitor the program. It does provide information to programs aiming to decrease risky behavior, however this particular behavior is not addressed by the program activities so it is irrelevant. In the second example, the indicator might be right. If you are distributing stoves, measuring the % of households with those stoves might work. However, do you know how many stoves were in households before you began? Are they the stoves you are distributing? Better might be an output indicator of how many stoves you distributed as part of your program within the last X months. In a nutrition program using community based distributors to improve access to oral rehydration therapy for children with diarrhea, an indicator measuring distribution of ORS by clinics would not be relevant. Although it might be interesting to know in relation to their program activities, it does not contribute to program monitoring.

Factors to Consider When Selecting Indicators
Logical – are they linked to framework? Programmatic needs – do they get you the information you need for decision making? Resources – can you afford to collect it? External requirements – do you need it for government, donor, headquarters? Data availability – can you get the data you need for both numerator and denominator? Standardized indicators – can you compare across programs/countries? Is there a “gold standard” for this indicator? Are they in the PHE Indicator Guide? In an ideal world, indicators judged to be the highest quality and most useful would be the ones selected and used to monitor and evaluate program activities. However, in the real world and in field settings, many other factors may intervene. Links to the program activities, as shown through frameworks is important, as are the needs of the program for decision making. Ideal indicators may not be practical; the feasibility of certain indicator designs can be constrained by data availability, financial and human resources. Requirement and needs of donors, the government, organization headquarters, and others may need to be given priority. Some examples of these considerations are: Availability of Data: Some data may be considered ‘privileged’ information by agencies, projects, or government officials. Data may be available only on aggregated levels or already calculated into indicators that may not be the ideal indicators for your program or activities. Resources: Ideal indicators might require collecting data to calculate an unknown denominator, or national data to compare with project area data, or tracking lifetime statistics for an affected and/or control population, etc. The costs of collecting all of the appropriate data for ideal indicators typically is prohibitive. Human resources and technical skills, particularly for evaluation, may be a constraint as well. Programmatic and External Requirements: Indicators may be imposed from above by those not trained in M&E techniques. Reporting schedules may not be synchronized (e.g. fiscal year vs. reporting year). Different stakeholders priorities may diverge. Discussion: In your experience, what factors other than desire to select the best and most appropriate indicators, has affected the selection of M&E indicators? Give specific examples.

Operationalizing Indicators
To operationalize an indicator is to identify how a given concept or behavior will be measured Challenges: Subjective judgment Local conditions Unclear yardsticks Available data In a best-case scenario, indicators will be conceptually clear, lending themselves to easy, unequivocal, unambiguous measurement. For example, the number of persons completing a given training course, in a given year, by category of personnel (e.g., park guard, nurse, gender). Every effort should be made to design and define indicators so that their operationalization will be as transparent as possible. There are a number of challenges in operationalizing indicators and they tend to fall into one of these categories: subjective judgment, local conditions that need to be understood or maintained, and using unclear units, terms, or yardsticks for calculating indicator values. Let us go over each of these. Measurement of some indicator terms may be subjective. For example, quality; leadership; improvement; networking; advocacy. Often, the indicator itself could be re-designed to specify a more precise, singular dimension of a result. However, sometimes subjectivity cannot be avoided, in which case very precise definitions must be agreed upon and used in every M&E cycle. Local conditions may affect measurement of an indicator. For example: new acceptors can only be counted according to the kinds of records that distributors are willing to keep on a continuing basis. If you want to define new acceptors as only the first time a woman begins any (modern) contraception (rather than a new acceptor as any client who starts any method at any time from any distributor), you will need very good, detailed, and cross-referenced data. Otherwise, it may not be feasible to identify those kinds of “new acceptors.” You will need to tailor your definition to the data that are locally available. Indicators may be defined without a clear yardstick. For example “cost of one month’s supply of fertilizer” does not make clear how cost will be defined? Per person? As an average? Should the average cost be weighted by the proportion of the surveyed population using each different method? For which month? Costs may vary over the course of a year, especially in areas of high or seasonable migration. What about unstable or fluctuating currency rates. All these issues need to be addressed in what we will call the indicator metric, agreed upon by relevant stakeholders, and recorded and used in M&E planning, implementation, and interpretation and use of the results.

CHALLENGES FOR PHE INDICATORS

Indicators for M&E of PHE programs
This Guide was created to address several key challenges in terms of quality M&E efforts in the PHE community. First, this Guide is a tool that can help you measure the results and rewards of integrated PHE projects and demonstrate the benefits of the PHE approach to achieving larger development goals. Second, no single implementing organization can be a technical expert in all three fields, P, H, & E. So, this easy-to-use guide provides you with the most important and trusted indicators across the population, health and environment fields. This guide was developed in consultation with current leading PHE practitioners, and the included indicators are those that have been tested successfully in past PHE projects. This manual facilitates improvements in M&E of PHE programs by making indicators available and easy for you to use– information and technical resources at your fingertips! As a result of this effort, the Guide includes: A broad set of common PHE indicators to monitor and evaluate PHE programs at both program and population levels – giving you an invaluable tool to plan and implement your M&E plan; and Standardized definitions of indicators and terminology that can be used across PHE programs, countries, and donor agencies – giving us a way to communicate and aggregate between projects, organizations, and practitioners.

PHE Indicators Population Indicators
Reproductive health; family planning; births Health Indicators Vaccines; safe water; bed nets Environment Indicators Legally-protected area; community-based NRM; species richness Integration Indicators Linked messages; collaboration; awareness Value-Added Indicators Gender; governance; economic development; youth [OPEN THE BOOK AND EXPLORE THE SUMMARY LIST OF INDICATORS ON PAGES ALLOW PARTICIPANTS A FEW MINUTES TO LOOK THROUGH THE GUIDE IF THEY HAVE NOT ALREADY.]

Each Indicator Includes:
Description/definition Timing of indicator – measurement Activities (processes/inputs) or Results (outcomes/impact) Calculation of the measure Counts (number of guards trained and posted; number of trees planted) Percentages, rates, ratios (contraceptive prevalence rate; % of planted trees surviving) Purpose of the indicator Data source and disaggregation Frequency of data collection Strengths and weaknesses [PICK ANY INDICATOR TO USE AS AN EXAMPLE HERE.]

Integration Indicators
Help us measure our integrated PHE efforts across programs Demonstrate commitment to providing diverse PHE messages and services Show the importance of PHE issues to the communities and countries where you work Help demonstrate community support, ownership, and sustainability of PHE Could indicate the cost-effectiveness of these cooperative efforts We touched on these indicators earlier in the training, and we will practice using them in this module. Again, although many of the indicators in this guide were tested and used in the field, there are no standard measures of integration, in part because we integrate between sectors and link our programs in unique ways. The integration indicators in the book are an attempt to begin to standardize across measurement of these aspects of our programs. The integrated indicators in the book provide partial answers for these measurement questions. [READ SLIDE]

Data Sources: Project records Time frame: Monthly, quarterly
Integration Indicator #3: Number and frequency of PHE sessions provided in the target community (p. 108) Definition: Educational sessions should be on topics related to integration of population, health and environment issues rather than sessions on specific and single-sector topics. These sessions could occur in any context such as presentations to local officials, a community theater presentation, or as part of a community educator’s typical work schedule. Purpose: This indicator captures the extent to which the project is educating the target population on the linkages between humans and the environment. Counting the number of sessions can provide an indication of project progress. Data Sources: Project records Time frame: Monthly, quarterly Data Collection Considerations: Sessions may include community visits, outreach home visits, educational activities, videos, or communication programs with integrated approaches. Strengths & Limitations: This indicator is easy to collect with good project record-keeping; however it does not give indication of whether the target audience received the message or of their understanding and acceptance of the linked message. Let’s look through an example of an integration indicator. This is an indicator that could work for many integrated programs that do multi-sector educational outreach, and it is relatively easily to collect and report. [ASK PARTICIPANTS TO OPEN THEIR GUIDES AND READ THROUGH SLIDE. OR, HAVE PARTICIPANTS HELP YOU READ THROUGH THE SLIDE OR DESCRIPTION IN THE GUIDE. IF ONE OF THE OTHER GUIDE INTEGRATION INDICATORS WORKS BETTER FOR YOUR ORGANIZATION, USE A SECOND EXAMPLE.] [START A DISCUSSION HERE USING THIS INTEGRATION INDICATOR. POSSIBLE QUESTIONS INCLUDE: WHAT ACTIVITIES COULD YOU MEASURE TO INTEGRATED PHE EFFORTS? WHAT INDICATORS COULD YOU USE TO DEMONSTRATE THE VALUE OF YOUR INTEGRATED PHE ACTIVITIES AND EFFORTS? HOW CAN YOU BEST MEASURE YOUR INTEGRATED APPROACH FOR FUNDING AGENCIES AND POLICY MAKERS?]

Value Added Indicators
Demonstrate what was achieved outside of the initial program plan Show how the unique aspects of PHE programs create larger changes than single sectors programs Illustrate how PHE programs are more than the sum of their parts Offer a way to “gain credit” for the additional changes we create in gender, governance, livelihoods, and youth Let’s return to the issue of why PHE is a value added approach, this time in the context of indicating these unique contributions through our selection of indicators. As we discussed earlier, our integrated approach and innovative programmatic models have potential to contribute to positive gains in sectors outside of P,H, or E. Specifically, our programs frequently “add value” by making positive changes in the areas of gender, governance, youth, or livelihoods. [QUESTIONS FOR GROUP DISCUSSION ABOUT THE IMPORTANCE OF VALUE ADDED INDICTORS] How do your integrated efforts make more sustainable long-term changes for the communities and areas in which you work? Where do you note the important contributions that your programs make that are not in your original program or monitoring plan? How can we show donors that these programs are valuable? [THIS DISCUSSION AND EXAMPLES FOCUS ON GENDER. THERE ARE MANY VALUE ADDED INDICATORS THAT WORK ACROSS PROGRAM AREAS, AND YOU SHOULD LOOK THROUGH THE GUIDE FOR EXAMPLES THAT WORK FOR YOUR SPECIFIC CONTEXT. ADAPTATIONS THAT MIGHT WORK FOR YOU INCLUDE: IF YOUR PROJECT OR ORGANIZATION FOCUSES ON YOUTH EMPOWERMENT, CONSIDER MODIFYING THESE SLIDES AND USING VALUE ADDED INDICATOR 7 OR 11 FROM THE GUIDE. IF YOUR PROJECT OR ORGANIZATION FOCUSES ON LIVELIHOODS, CONSIDER MODIFYING THESE SLIDES AND FOCUSING ON VALUE ADDED INDICATORS 2, 3, 17, OR 18. IF YOUR PROGRAM OR ORGANIZATION FOCUSES ON FOOD SECURITY, CONSIDER MODIFYING THESE SLIDES AND FOCUSING ON VALUE ADDED INDICATORS 5, 6, OR 15. IF YOUR PROGRAM OR ORGANIZATION FOCUSES ON NATURAL RESOURCE MANAGEMENT AND GOVERNANCE, CONSIDER MODIFYING THESE SLIDES AND FOCUSING ON VALUE ADDED INDICATORS 1, 7, 8, 9, OR 16.] [THIS IS AN ADDITIONAL LOCATION IN THE TRAINING COURSE WHERE YOU COULD DISCUSS HOW YOUR SPECIFIC PROGRAM PROVIDES ADDITIONAL VALUE FOR THE AREAS IN WHICH YOU WORK RATHER THAN SEPARATE P-H-E SECTOR EFFORTS.]

Data Sources: Project records Time frame: Annually
Value added Indicator #8: % of leadership positions held by women on community NRM committees (p. 129). Definition: A “leadership position” is any position where the chosen person has commanding authority or influence. Purpose: An NRM committee may make decisions regarding tasks completed by women; therefore this indicator measures the ability of women to have a decision-making role in committee plans, actions, and control of resources. Increased participation of women may lead to decreased gender inequality. Data Sources: Project records Time frame: Annually Data Collection Considerations: Qualitative interviews with women may assess their perceptions of involvement in the committee and obtain exact details of their leadership responsibilities. Strengths & Limitations: Women involved in decision-making for the NRM may also have influence in education, health, and family planning decisions. However, there may be cultural factors that prevent female leaders from speaking up; there is no equality if women play only a ceremonial role. Example: Indicator 8 – Value Added [READ THROUGH SLIDE] Let’s look closer at the purpose component to understanding why this is value added. In this case, it’s potential added value is in the positive effects it could have on gender relationships. As noted in the Guide, an NRM committee made up of only men may make decisions on issues that affect primarily women, such as regulating activities typically completed by women such as firewood collection, water collection, or selection of areas for planting. Therefore, this indicator could reflect gender equity and the ability of women to have a decision-making role in committee plans, actions, and control of resources. Increased participation of women on these community-based NRM committees may lead to overall gains in the balance of power between men and women in the community as a whole. Another example of a value-added indicator in a PHE project is “net dollar value of socially-marketed products sold.” The PHE project may intend to increase the use of condoms by providing commodities to local merchants and providing merchants with sales training, for example. In the process of reaching the goal of increasing use of condoms, the project may also diversify the livelihood of that merchant by providing an alternative or supplementary stream of income. The diversification of livelihood for that merchant is a value-added result. [OTHER GENDER-RELATED VALUE ADDED INDICATORS ARE #4, 8, AND 13.]

Data Sources: Project records Time frame: Annually
Value Added Indicator #10: Number of fuel-efficient stoves distributed (page 133). Definition: Fuel-efficient stoves are enclosed stoves that often employ an elbow shape to provide a combustion chamber and insulation that increases the heat available to cook food. Purpose: Switching to fuel-efficient stoves can have direct impacts on both forest and human health by limiting wood collection and by minimizing human exposure to pollutants. Data Sources: Project records Time frame: Annually Data Collection Considerations: A standard form should be used to keep track of these data. The form should include the dates of distribution, total # of stoves distributed, a list of recipients, and the village or community name. Strengths & Limitations: Data on the number of stoves distributed should be easy to obtain and track over time. However, the distributed stoves only have an impact if they are used. Let’s look at another value added indicator as well. Again, a better understanding of the purpose can help us clarify what it means to be value added: [READ THROUGH SLIDE. TEXT FROM GUIDE PAGES ] Firewood Collection and Gender-Based Violence “Fuel-efficient stoves reduce the consumption of firewood, thereby conserving more of the natural environment and decreasing smoke in the cooking area which can lead to respiratory infections. However, they can also serve to protect women and girls by reducing their time spent searching for and collecting firewood. In many regions of the world, women and girls collect firewood and fuel used for cooking or for income. In some regions, especially among refugee and displaced populations, women and girls are at risk of gender-based violence (GBV), including rape or physical assault while collecting firewood for their families. Female refugees interviewed by Refugees International cited the threat of violence while collecting firewood as one of their top concerns. In households with less need for firewood, women and girls will spend less time collecting firewood and also may not need to travel as far. Walking far from home to find firewood increases the risk of GBV because women may have to walk in isolated, unknown areas or go near military posts or checkpoints where assaults are more likely. Additionally, women can be trained to construct and maintain fuel-efficient stoves. This alternative income-generation activity may further reduce the risk of GBV by reducing the need to sell firewood. Fuel-efficient stoves can enhance the lives of women and girls in several other ways, including reducing risk of respiratory infections, improving their natural environment, freeing time for other income-generating activities.” [IF YOUR PROGRAM TARGETS YOUTH, YOU MIGHT WANT TO USE VALUE ADDED INDICATOR #11 OR CONSIDER ANOTHER INDICATOR FROM THE GUIDE THAT MAKES SENSE FOR YOUR PROGRAM OR PROJECT.]

Special focus: Integrating Gender
Gender sensitive indicators Women’s participation in program activities Women in leadership roles Young women’s decision making, etc. Collect program-related information among men and women Hold focus groups, conduct informational interviews with men and women Collect household information from men and women Sex-disaggregated data – report information separately for men and women (participation, leadership, etc.) when possible Gender can be incorporated from the program plan through to the program outcomes. It might be an intended focus of your program or part of the secondary, positive effects that your program brings. Either way, and as discussed before, gender merits special attention especially for PHE. To that end, here are some simple ways that you could collect information on gender and simultaneously “gain credit” for the positive effects your program has on women. For example, you could collect information separately for men and women, keep track of participation by sex, note the leadership of your programs, and keep records of who attends meetings by gender. Many of these ways of collecting and recording this gender-specific information are easy and require little to no additional work. As you are likely already reaching out to include women, it is best for you to report it and gain credit for your contributions. [READ SLIDE] [HERE YOU COULD CHANGE THIS TO SPECIAL FOCUS TO YOUTH, GOVERNANCE, INCOME, OR WHATEVER OTHER SPECIFIC SLIDE OR EMPHASIS WOULD MAKE SENSE FOR YOU AND YOUR PROGRAM. IN THIS EXAMPLE, GENDER IS HIGHLIGHTED. YOU COULD EASILY CHANGE THE SLIDE TO DISAGGREGATE BY AGE OR ABOUT INCLUSION OF YOUNG ADULTS. THE MAIN POINT IS THAT GAINING CREDIT FOR YOUR VALUE ADDED IS NOT COMPLICATED, BUT IT IS IMPORTANT. WE NEED TO HIGHLIGHT OUR ADDITIONAL ACHIEVEMENTS, COLLECT INFORMATION ON THESE TYPES OF INDICATORS, AND DISSEMINATE OUR FINDINGS TO ADVOCATE FOR MORE SUPPORT FOR THESE EFFORTS. WE CANNOT PROMOTE INTEGRATION WITHOUT EVIDENCE OF OUR UNIQUE CONTRIBUTION!] 102

Common Indicator Metrics
Counts Number of providers trained Number of condoms distributed Calculations: percentages, rates, ratios % of parks with trained guards Contraceptive prevalence rate, denominator and numerator! Index, composite measures Quality index comprising the sum of scores on separate outcome indicators Thresholds Presence, absence Pre-determined level or standard Perhaps the most important part of what comprises an indicator is the metric. The metric is the precise explanation of the data and the calculation that will give the measurement or value of the indicator. In other words, it specifies the data that will be used to generate the value, and how the data elements will be manipulated to come up with a value. Defining good metrics is absolutely crucial to the usefulness of any M&E plan. A good metric clarifies the single dimension of the result that is being measured by the indicator. A good metric does this in such a way that each value measured for the indicator is exactly comparable to values measured at another time. Indicators can have a number of types of metrics. They can be simple counts of things (for example, the number or providers trained or the number of stoves distributed), or they can involve calculations (for example, the proportion of park with trained guards, contraceptive prevalence rate, % of households participating in alternative income generation). They can also be more complex, such as an index comprising of the sum of scores on six quality outcomes. These are just some examples of types of metrics used for indicators. [THE PURPOSE OF THIS SLIDE IS TO BE SURE PARTICIPANTS UNDERSTAND WHAT THE TERM “METRIC” REFERS TO AND TO HELP THEM RECOGNIZE COMMON FORMATS OF INDICATORS. THIS SLIDE DOES NOT INTEND TO PRESENT RECOMMENDED OR GOOD INDICATORS, OR TO COVER ALL TYPES OF METRICS USED IN CALCULATING INDICATORS.]

The Right Information for the Right Indictor: Indicator Metrics
Indicator 1: (p.49) Percent of program staff trained to work with or provide reproductive health services to adolescents What does “trained” mean? What information do you need to collect this? Indicator 2: (p.87) Percent of trees planted that survive Numerator: what would this be? Denominator: what would this be? What does “survive” mean? So, now that we know about what indicators are, how do we get the right information for the right indicator? Behind an indicator metric is information important to its correct calculation. It is important to understand what goes into an indicator before we move on to discussing how they are selected or developed. Here are two examples: In the first example, the indicator is the number of providers trained under a given program aiming to improve the quality of reproductive health care for adolescents. For this example, let’s assume that part of the program includes training of providers. It is important to note that “providers” are defined as any clinician (meaning doctor, nurse, or medical assistant) providing direct clinical services to youth clients seeking reproductive health services at public health facilities. Note that any providers working at private facilities are not to be included in this indicator. If the terms weren’t defined, the indicator could be counted or interpreted differently than intended. Also note that in order to be counted for this indicator, we will note that the clinician has to have attended all five days of the training course. This means that this information must be collected at the time of the training workshops. To measure whether a clinician was trained, what type of tool would you use? This is where a pre- and post-test might be very useful. It will help you gage whether your training achieved its goals – M&E should happen all the time – meaning that every activity should have a way to M&E its effectiveness through a reporting and verification process. In the second example, the indicator is % of trees planted that survive. What information would you need for this indicator? How would you define survival? [NUMERATOR: # OF SURVIVING TREES AT X MONTHS OR YEARS] [DENOMINATOR: # OF TOTAL TREES PLANTED DURING THAT SAME PERIOD] [THE PURPOSE OF THIS SLIDE IS TO BE SURE PARTICIPANTS UNDERSTAND WHAT GOES INTO A METRIC AND TO INTRODUCE THE TERMS “NUMERATOR” AND “DENOMINATOR” INTO THE DISCUSSION. THE KEY MESSAGES ARE AS FOLLOWS: 1) FOR COUNTS, SPECIFY WHAT/WHO QUALIFIES TO BE COUNTED AND WHEN; 2) FOR PERCENTAGES, ALWAYS SPECIFY THE NUMERATOR AND DENOMINATOR.]

Always Specify the Details!
For example: Environment Indicator #8 – “improved technology adoption” (p.92) How would you define “adopt” ? [YOU MAY CHANGE THE EXAMPLE IF YOU WISH] For this example, let’s look on page 92 of the Guide about whether farmers/fishers adopt improved agricultural/marine practices. In a discussion about important details, how would you define the word, “adopt”? [ASK:] Is “adopt” if they use new practices 90% of the time? What are the “new” practices? Which specific ones? Only project ones or any new practice? What if they try them for one month and then return to old practices?

Sources of Indicators: Pre-defined Indicators
From past years of the program From related or similar programs From lists of global or recommended indicators MEASURE Evaluation PHE Indicators – (Guide) Other resources listed in the training course, slides 150+ Given all the characteristics a good indicator should have, and since it is not necessary to re-create the wheel for every M&E plan, keep in mind that there are pre-existing sources of indicators. An obvious source is from past years of a program. Because monitoring program progress over time is a key objective of an M&E system, it is imperative to continue to use the same indicators if they meet the characteristics of a good indicator as described previously. However, improvements to indicators should be made when necessary. If different indicators are used over the life of a program, a decision must be made to either stop collecting/reporting the old indicator and begin with the new, or to continue with the old and add the new, noting the difference in the M&E plan or in reports. Another source is related or similar programs. The ability to compare your program outcomes with others may be a desirable feature, so when indicators used in other programs would also be useful in yours, you may want to consider those for your M&E plan. There are lists of global indicators that should be consulted. For us in PHE, this is the Guide that we are using for this training. Some other international organizations produce guides for M&E and include recommended indicators. Often, M&E and subject-matter experts take part in working groups led by these organizations to compile lists of indicators, and these guides are typically updated from time to time. In some cases, these guides also include data collection instruments that have been field tested. Many of the resources where you could find indicators are listed in the end of these training modules.

Adding Indicators to Logic Model
Input Activity/ Process Output Outcomes Impact Quantifiable resources going in to your activities – the things you budget for. 1. What you do to accomplish your objectives? Immediate results from your activity – people trained, services provided Longer-term change in knowledge, attitude, behaviour, etc. related to program goal Long-term, population level change. Can relate to a program or organization’s vision/mission statement Indicators (example) How do you measure this? Lets quickly review what is an input, activity, output, outcome, and impact before we add indicators. [READ TEXT ON TOP TWO ROWS] Now, let’s look at the bottom row of indicators. Reading across: how would you measure your progress from inputs through impacts? Important note: Inputs, activities, and outputs are generally counts – what is the immediate result of your activity? Outcomes and impacts area generally percentages or proportions – what are the longer term changes in knowledge, attitudes, and practices that result from your activities and outputs?

Logic Model: Family Planning Activity
INPUT Human and financial resources Demand for FP in the community PROCESS Educate women about the advantages of modern method use Distribute FP methods in the community Train program staff in providing FP information and methods OUTPUT Increased interest in FP Sessions held in community about family planning methods Clinic staff trained in FP method counseling OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP IMPACT Increased contraceptive prevalence This logic model should look familiar from Module #4 – frameworks, part 2. Now, we are going to take this same example and add indicators to help us indicate whether we achieved our intended outputs and outcomes. You may also collect information about inputs and processes, but we are most concerned with whether you implemented your program and have achieved the intended outputs – immediate results of your activities – and outcomes – longer term changes in KAP. [IF YOU WANT TO REPEAT TEXT AS A REMINDER: IN THIS EXAMPLE, THE INPUT IS THE HUMAN AND FINANCIAL RESOURCES TO INCREASE FAMILY PLANNING USE. THE PROCESS, OR PROGRAM ACTIVITIES RELATED TO THIS, WILL BE THE DISTRIBUTION OF METHODS, EDUCATING WOMEN, AND TRAINING PROGRAM STAFF. THE ANTICIPATED OUTPUT IS THAT THE BROCHURE IS DISTRIBUTED TO CLIENTS OF THE FACILITIES. THE OUTCOMES ARE INCREASED INTEREST IN FP, HOLDING EDUCATIONAL SESSIONS, AND HAVING STAFF TRAINED. THE OUTCOME – SHORT TERM CHANGES IN KNOWLEDGE, ATTITUDES, OR PRACTICES, ARE INCREASED ACCESS, CONDOM USE, AND MALE PARTICIPATION WHICH WILL LEAD, IN LONGER TERMS, TO INCREASED CONTRACEPTIVE PREVALENCE IN THE COMMUNITY.] 108

Adding indicators: Family planning activity
INPUT Human and financial resources Demand for FP in the community PROCESS Educate women about the advantages of modern method use Distribute FP methods in the community Train program staff in providing FP information and methods OUTPUT Increased interest in FP Sessions held in community about family planning methods Clinic staff trained in FP method counseling OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP IMPACT Increased contraceptive prevalence [YOU WILL COMPLETE THIS ACTIVITY AS A LARGE GROUP IN PREPARATION FOR COMPLETING THE SAME TYPE OF ACTIVITY IN PARTNERS SHORTLY] This is the exact same model, now with room for you to add indicators. For this example, and for this training, we will focus on output and outcome indicators. These are most important for your stakeholders, donors, and partners to show results. Using your Guide and the module we just completed on indicators, can anyone provide a possible indicator for one of these three outputs? How would we measure and show success here? Keep in mind that there are many right answers. Also, keep in mind that value added could be an output, outcome, or impact. As an example, how could gender indicators be measured as value added in these outcomes, outputs, or impacts? Bonus points for partners that find added value gender-related indicators. [TAKE ~10 MINUTES TO BRAINSTORM SOME INDICATORS BEFORE THE NEXT SLIDE] [AGAIN, IF YOUR FOCUS IS YOUTH OR LIVELIHOODS, CONSIDER HOW YOU COULD FIND A “VALUE ADDED” INDICATOR THAT WOULD WORK FOR YOU.] Indicators: INPUT PROCESS OUTPUT OUTCOME IMPACT 109

Adding Indicators: Family Planning Activity
INPUT Human and financial resources Demand for FP in the community PROCESS Educate women about the advantages of modern method use Distribute FP methods in the community Train program staff in providing FP information and methods OUTPUT Increased interest in FP Sessions held in community about family planning methods Clinic staff trained in FP method counseling OUTCOME 1. Increased access to FP services 2. Increased condom use 3. Increased male participation in FP IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence IMPACT Increased contraceptive prevalence OUTCOME % of women who were clients of community-based FP distributor last year (Pop #2) # of new acceptors of FP methods (Pop #6) % of men who approve of FP for themselves or their partners (Value added #13) Indicators: Together, let’s look at some of the possible indicators that we could use. Note that the outputs are generally counts and that the outcomes are generally percentages. [READ SLIDE] Notice that there are many from the Guide that would work and many others that would also work well. Can someone think of a possible integration indicator that might work here? Perhaps suggesting a small change in the programmatic approach? Are there other value added indicators that might work here? [VALUE ADDED 4 OR 13 MIGHT WORK HERE.] OUTPUT % of program staff trained to provide FP information to youth (Pop # 1) # of educational session held in the community about FP methods INPUT PROCESS IMPACT Contraceptive prevalence rate (Pop #7) 110

Logic Model: Diarrhea Prevention Activity
INPUT Human and financial resources Demand for disease intervention in community PROCESS Train park guards in diarrheal disease and use of oral rehydration salts Train local community members in safe water storage Form natural resource management committees in 10 communities near park OUTPUT (ORS) packets distributed Community members trained in safe water storage NRM committees formed OUTCOME 1. Reduction in acute diarrhea cases among children 2. Increased safe water storage 3. Involvement of NRM committees in promotion of safe water IMPACT Reduced morbidity /mortality from diarrheal disease among children Here is another example of a logic model. This one represents a diarrhea prevention component for an integrated program aimed at reducing childhood disease due to unsafe water. Imagine this project takes place in communities with high diarrhea incidence and that the community is located within a national park buffer zone. This component is both Health and Environment because the community is located in a national park and park guards are involved in outreach activities. The inputs, processes, outputs, outcomes, and impacts are only a piece of a larger puzzle and are not intended to be perfect or complete. [READ THROUGH COMPONENTS ON SLIDES.] 111

Adding Indicators: Diarrhea Prevention Activity
INPUT Human and financial resources Demand for disease intervention in community PROCESS Train park guards in diarrheal disease and use of oral rehydration salts Train local community members in safe water storage Form natural resource management committees in 10 communities near park OUTPUT (ORS) packets distributed Community members trained in safe water storage NRM committees formed OUTCOME 1. Reduction in acute diarrhea cases among children 2. Increased safe water storage 3. Involvement of NRM committees in promotion of safe water IMPACT Reduced morbidity /mortality from diarrheal disease among children Now, let’s look at an example set of activities from an integrated PHE program to reduce diarrhea in communities within a national park. Just like the other example, we need to add indicators. You can see here that the three example activities (processes) combine efforts across the health and environment sectors. There are also possibilities for “value added” results as an outcome of these efforts. Working in pairs, now you’ll come up with six indicators – three outputs and three outcomes – including at least one value added and at least one integration indicator (although you may have more). Again, there are many right answers. [GIVE THEM ~20 MINUTES; ~10 MORE FOR GROUP DISCUSSION. YOU MAY PASS OUT BLANK PAPER FOR PARTNERS TO WORK ON OR YOU MAY PASS OUT ENOUGH COPIES OF THE POWERPOINT SLIDE “BLANK LOGIC MODEL_WITH INDICATORS” FROM THE TRAINING ACTIVITY FOLDER FOR PARTNERS TO USE FOR THIS ACTIVITY.] OUTPUT 3 HERE – 2 from the GUIDE *including at least one value added Integration indicator??? Indicators: OUTCOME 3 HERE – 2 from the Guide *including at least one value added! Integration indictor? INPUT PROCESS IMPACT 112

INPUT Human and financial resources Demand for disease intervention in community PROCESS Train park guards in diarrheal disease and use of oral rehydration salts Train local community members in safe water storage Form natural resource management committees in 10 communities near park OUTPUT (ORS) packets distributed Community members trained in safe water storage NRM committees formed OUTCOME 1. Reduction in acute diarrhea cases among children 2. Increased safe water storage 3. Involvement of NRM committees in promotion of safe water IMPACT Reduced morbidity /mortality from diarrheal disease among children Indicators: INPUT PROCESS OUTPUT # of NRM committees formed in the 10 communities near the national park # or ORS packets distributed (health 3) Number of educational sessions held about safe water sources, storage, etc. % of households with access to improved water source (health 7) Guide Integration indicators # 2, 3, 4, or 5 may work!!!! OUTCOME % of households with access to n improved water source (health #7) % of communities in target area with community based (NRM) plans (Env’t 1) Oral rehydration therapy use rate (health 13) % of households storing drinking water safely (health 11) Value Added Indicators: Guide value-added indicators #1, 7,or 8 might work! IMPACT [BEFORE SHOWING THIS SLIDE, ASK PAIRS TO PRESENT THEIR OUTPUT AND OUTCOME INDICATORS. THEN, PRESENT THIS SLIDE OF POSSIBLE SOLUTIONS.] Here are some of the possible answers. This list is NOT exclusive of other options. Remember: you do not need more than ~1 indicator for key activity, so choose the best one. More indicators does NOT equal better measures of success! 113

Indicator Reference Sheet
Definition: detailed documentation for each indicator Basic information Description Plans for data collection Plans for data analysis, reporting, and review Data quality issues Performance data table (baseline and targets) [MAKE SURE EVERYONE PULLS OUT THE PRINTOUT OF THE WORD DOCUMENT “INDICATOR REFERENCE SHEET_GUIDED”] You have an example of an indicator reference sheets in your handouts. Please find that handout and follow along as I describe each section. The section titles are listed on this slide. As you can see, the Indicator Reference Sheet is divided into six sections: Basic Information – such as the indicator itself, when it was established and last reviewed, and how it links to the framework (e.g. to which strategic objective it pertains). Description – definition, unit of measure (number, proportion, average), method of calculation, and the justification and how it will be used in program management. Plans for Data Collection – method, source (survey, program records, etc.), frequency and timing, estimated cost for data collection, responsible organization or individuals for data collection, and where the data will be stored Plans for Data Analysis, Reporting and Review – this includes how it will be reported (in a table or graph, for example). Data Quality Issues – known limitations and any plans to remediate them. This can also include discussion of how these issues were weighted in selecting the indicator. Performance Data Table – this should contain information with the years of the program and the baseline and target values for each year. If the indicator is to be reported every two years, for example, this would be reflected in this table as well.

Population-Based Indicator Reference Sheet
Core Indicator Relating to Key Result (Objective): Required by Projects Conducting Surveys Indicator Description Indicator: Contraceptive Prevalence Rate (Met Need) Definition: Percentage of women of reproductive age (WRA) who are married or in union using (or whose partner is using) a modern method of family planning Unit of Measure: Percentage How to Calculate: NUMERATOR: No. of women age 15–49 years who are married or in union who are not pregnant (or unsure) AND who report using (or whose partner is using) a modern method of family planning. DENOMINATOR: Total number of women age 15–49 years who are married or in union included in the survey Indicator = (Numerator/Denominator) * 100. Important Background Characteristics to Consider When Assessing This Indicator: At minimum, you are encouraged to examine this indicator by age group. Modern methods: The following methods are usually counted as modern methods in the indicator: female sterilization (tubal ligation), male sterilization (vasectomy), pills, IUD, injections, implants (NORPLANT), condoms, female condom, diaphragm, and foam/jelly and LAM. The Standard Days Method (SDM) is also in the process of being designated as a modern method of FP. What It Measures: Population coverage of current family planning use Important Notes: The conventional indicator is limited to women in union/married. Data Collection Data Source: Population-based household survey (Flexible Fund Family Planning Survey) Data Collection Method: To obtain the CU/CPR, please refer to the Flexible Fund Family Planning Survey for model questions required to construct the indicator. The most recent version is found on the website: Frequency/Timing of Data Collection: Baseline, midterm (if applicable), and final evaluation Proposed Data Use: The overall purpose of this indicator is to assess whether all your project activities are contributing to the ultimate objective of increased contraceptive use. If the CU or CPR is lower than expected, consider all the possible factors that contribute to end use (e.g., knowledge/interest, quality of care, access). [MAKE SURE EVERYONE PULLS OUT THE PRINTOUT OF THE WORD DOCUMENT “INDICATOR REFERENCE SHEET_GUIDED”]

Setting Indicator Targets: Useful Information Sources
Past trends Client expectations Donor expectations Expert opinion Research findings What has been accomplished elsewhere It is important that targets be established for performance indicators in order to assess how much change or improvement has been achieved. Setting performance targets can be challenging. Collaborating with others who are knowledgeable about the local situation, and who understand what level of accomplishment can be expected is key to setting targets that can realistically be achieved, given the available resources, time frame and the environment. There are various approaches that you should consider when trying to determine appropriate target levels: 1) look at past trends and project them into the future; 2) consider client and donor expectations; 3) consult experts to determine appropriate target levels; 4) review the literature to find out what has been accomplished elsewhere; 5) consider how long it takes to see change. For example, how long will it take for your education and training programs to affect the fishing behaviors of your target population? Equally important is documenting the rationale for the targets so that you can refer to this reasoning when trying to analyze or interpret actual data later on. Note: the indicator reference sheet also requires that you describe the rationale that was used in setting targets under “Notes on Baselines/Targets.”

Common Pitfalls in Indicator Selection
Indicators not linked to program activities Using outputs as outcomes Poorly defined indicators Data needed for indicator is unavailable Indicator does not accurately represent intended outcome Too many indicators in P, H, and E Let’s go through these one by one.

Pitfalls with Selecting Indicators
Indicator not linked to program activities IR: Expanded access to diarrhea treatment services Activities: Train providers in treating acute diarrhea Inappropriate Indicator: % of facilities with adequate conditions to provide care Better indicators: # of clinicians trained, # of facilities with a trained provider, % of clinicians with 100% on post-test One of the common pitfalls in selecting indicators is to choose one that the program activities cannot affect. This is an example of something that did occur. There was a program that planned to train health care providers in diarrhea treatment in an effort to reduce mortality from diarrhea with rapid, correct treatment. The authors of this M&E plan reviewed some of the documents produced by WHO and selected some indicators that seemed important. Specifically, they wanted to report the proportion of health care facilities with adequate conditions to provide care. However, many things go into that proxy indicator, including supervision, supplies, and equipment availability, and the drafting of appropriate treatment protocols – several things that the program did not intend to address directly. In trying to use a global indicator, they overlooked the fact that it was not related to their program activities. A better indicator would be the # of clinicians trained or the # of facilities with a trained provider to address acute diarrhea in children. Ideally, indicators will provide feedback to the program so that changes can be made to activities to more effectively or efficiently reach program objectives. If an indicator is not something that is affected by program activities, not only is the indicator measuring something irrelevant, but it also is measuring something that cannot be useful for the program. How would you measure provider skills or knowledge? [PRE/POST TEST, ON-THE-JOB SUPERVISION, % OF CHILDREN APPROPRIATELY TREATED] The program is not aiming to affect facility conditions, only provider skills.

Using outputs to measure outcomes Problem: routine monitoring data (outputs) is available from households but outcomes are needed for reporting You collect: # of households that use your new crop rotation technique You need: % of households in target areas who use your crop rotation technique [LOOK AT ENVIRONMENT INDICATOR #9 FOR ANOTHER EXAMPLE OF THIS TYPE OF INDICATOR, PAGE IN THE GUIDE.] Often, our people-centered programs collect routine information from participating households. Some programs use community health workers, agricultural extension workers, or other types of community workers or volunteers to keep detailed project records on individuals or households. This collection of information may include data on health, family planning practices, agricultural practices, income, etc. Frequently, these project records and their data are compiled at the central office for routine monitoring. With a little extra work, it is relatively simple to turn these output indicators (results of your activities – like adopters of a specific intervention, attendance of a training, etc.) into outcomes indicators (longer term changes in knowledge, attitudes, and practices). Consider: You have routine data from households on an output, perhaps the # of households that use your specific crop rotation technique [CHANGE THIS TO MEET YOUR NEEDS]. You know this from your routine project records because the extension workers collect this information, and you are fairly certain that it is accurate. To turn that output indicator into a measure of a longer-term change in knowledge, attitude, or practice, and outcome, you need to know two major things: 1) how many households use your crop rotation technique (OR: how many people or households attended your training or were visited by your extension worker or were invited to your course, etc.); and 2) how many people or households, TOTAL, live in your project area? If you know those two things, then you can easily change your output into an outcome. Using the same example, you know that 0 households were using that technique when your program started. But, now after one year of intervention, 15 households use your crop rotation technique. Then, your output indicator for year one could be: “15 households use new crop rotation technique.” Now, lets say you want an outcome indicator of a change in knowledge, attitudes, or practices – in this case, the practice of crop rotation. Also, remember: outputs are usually counts and outcomes are usually percentages. To turn your output into an outcome, you need to know how many total households are in the target area. Now, you ask your extension workers to count the number of total households in the community (or use program records, district records, etc.). They determine (or the records show) that there are 145 households in the target area. Again, you already know that 15 households are using your new technique. And, now you know that there are a total of 145 households in the community. So, your outcome is 15/145 (~10%) use (adoption) of your new technology after one year! If you continue to collect the same information, which you should, you could measure that outcome again after three years. At three years, let’s imagine you have 67 households that are using the new crop rotation technique according to your project records. There are still 145 total households. Your new outcome is 67/145 (~46%). You can see from this that you could now report a 35% increase in new technology use over two project years. It’s relatively simple. [YOU COULD USE ANY EXAMPLE HERE, BUT THE BASIC POINT IS TO PROVIDE A REASONABLE EXAMPLE FROM YOUR PROGRAM OR IN YOUR TOPIC AREA THAT FOLLOWS THIS EXAMPLE OF TURNING ROUTINE MONITORING INTO OUTCOMES DATA WITHOUT TOO MUCH ADDITIONAL WORK, TRAINING, OR SKILL ON THE PART OF PROGRAM STAFF.] Routine monitoring data should not be reported as outcome data. Compile your routine data as well as information about your target area to turn outputs into outcome measures.

Indicator poorly defined Activity: radio campaign about PHE linkages or program components Inappropriate Indicator: % of population who have heard of X campaign Better indicators: % of population who can relate correct information about a PHE issue Another common mistake is to choose a vague indicator. We talked about this before. Although in many cases an indicator cannot be fully understood without seeing the precise definition of how it is calculated, it is possible to have a more readily interpretable indicator. In this example, an IEC campaign to increase PHE knowledge is to be monitored by the “exposure to PHE messages” of the population. In this case, it is not clear what type of knowledge is being measured or if people have any additional knowledge. A clearer indicator would be one that measured precisely the objective of the campaign, the proportion of the population that actually understand the PHE issue, or that find the PHE linkages important, or are aware of a specific issue. Hearing is not learning! The definition needs to provide more information. Campaign knowledge does not indicate what is desirable: knowledge that the PHE issues are important, what PHE issues are.

Data needed for indicator not available Data issue: Information on stock-outs may not be collected daily Inappropriate Indicator: % of days per quarter that service delivery points have stock-out of drugs Better indicators: % of service delivery points that had a stock out of drugs at some time during the last quarter Another common mistake is to select an indicator that relies on routine data without verifying that the data are available as you defined it. This is another “real world” example. A program working on drug supply selected an indicator for stock outs by “the number of days there were any stock outs.” When checking with the information collected at the health facilities, it was found that the health center only collected information about whether there were any stock outs in each quarter. This indicator was changed to account for the fact that information on stock outs was collected as a yes/no variable as well as to better represent the intended result of the program. It would be even better if it specified what the stock out was in: contraceptives, ORS, etc. If relying on routine data, indicator definition will depend on how data are collected.

Indicator does not accurately represent desired outcome IR: Expanded access to family planning methods Inappropriate Indicators: % of women using family planning methods Better indicator: # of new users of family planning; contraceptive prevalence rate; estimated CPR This example also comes from a real experience. In this example, the problem with the inappropriate indicators was not readily evident to the authors of the M&E plan. The details of each indicator’s definition must be considered. [ASK]: Does the % of women using family planning tell you anything about expanded access? What methods? If the percent changes, is it due to your program? How would you know? Better to consider indicators that reflect greater access such as number of new users, number of methods distributed by community-based distributors, number of new clients of community based distributors, etc. What does it mean if inappropriate indicators increase? Decrease? Do they reflect the desired program effect?

Too many indicators for P, H, and E Rules of thumb: One or two indicators per key activity or result (ideally, from different data sources) (e.g., training, committee building, new users of family planning) Try to include at least 1 “value added” indicator per objective area Try to include at least 1 “integration” indicator to represent each linked activity, partnership, or outcome Not more than indicators per programmatic area, for P, H or E A frequent question is “How many indicators should my program have?” The best answer is “It depends.” What does it depend on? Complexity of goals; costs of data collection and analysis; benefits and practical utilization anticipated for M&E results at the different conceivable levels. A reasonable rule of thumb might be one or two indicators per result, but that depends on how finely detailed your results may be. You should definitely have at least one or two indicators for every significant activity – remember, M&E’s purpose is to monitor program performance and evaluate outcomes and objectives! However, there is a tendency to go into overkill on indicators – especially for integrated programs with multiple important outcomes across diverse sectors. There is no 100% guide to selecting your best indicators, but to avoid reporting on too many, here are some hints: You do NOT need to collect indicators for every activity, just KEY activities. If three outputs lead to one outcome, consider collecting information or reporting information on just one of those outputs. There does not need to be a 1-to-1 correspondence between output and outcome indicators. You should report at least one, and maybe only one, indicator for each outcome. Indicators that will not be directly used for program management or make changes/improvements/scaling up to improve performance, effectiveness, or efficiency, are not a very sound use of program resources. Your data sources for your indicators should vary. Do not select indicators that all rely on project records or all rely on your partners for collection. It is wise to vary the data sources used for indicators, either secondary sources or your own data collection efforts, especially for key results. Any number of unexpected events can occur and disrupt an M&E plan such as budget cuts, delayed surveys, flood, fire, or water bottle leak! So, diversifying data sources is a good strategy to ensure some indicators can be tracked over the life of the project. Many PHE programs/organizations choose too many indicators because they are trying to cover three sectors completely. Again, one key outcome indicator of each key activity is usually enough and you do not need to report on every output! Also, there are no short cuts to program integration. The integration indicators in the guide help you demonstrate your multi-sector approach, but they cannot simplify the process of actually integrating! PHE programs need to select some indicators that demonstrate the value of a cross sectoral project by using value added indicators. Be thoughtful and creative about how you think your project goes beyond its intended impacts. Lastly, and we will look at this again in the next module, but if you do not have to report that indicator to a specific donor or information user, do not collect or report it! If no one wants to know that outcome, it might be an indication that you don’t need to collect it! There is no set formula for how many indicators. Consider your budget, stakeholders, and program plans – then, consider: is it reasonable? Feasible?

Indicator Pyramid Global Number of Indicators
Decreases Increases Global Compare countries Overview world-wide situation National/Sub-national Assess effectiveness of response Reflect goals/objectives of national/ sub-national response Again, trying to address the, “how many indicators should my program have?,” this indicator pyramid may be helpful. It shows that indicators at different levels (global, national or sub-national, or district or community) are used for different purposes and typically have different numbers of indicators. The higher up in the pyramid, the fewer the indicators you would consider. For most of our PHE programs, we would focus on the base of the pyramid at the district, community, or facility levels. District, Facility, Community Identify progress, problems, and challenges

Key Questions to Ask When Selecting Indicators
Do they meet programmatic needs? Will they give you useful information for decision making? Are they feasible considering time, money, and staffing? Do they match external requirements? Are the data available? How accurate are the data? Are they standard indicators used across projects, programs, countries? Are you collecting the information appropriately? In summary, here is a list of some questions you should think through and try to answer as you choose your indicators for your program. If you answer, “I don’t know,” or “no” to any of these questions about a specific indicator or group of indicators, you need to redefine your indicator or select a different one.

Small Group Activity Return to small groups
Return to the logic models from Module 4 Choose ONE of the logic models you completed Like the examples, fill in the indicators for your logic model. You may need to add additional detail for the output and outcome sections of your logic model before adding indicators. You must have at least 2 output and 2 outcome indicators. REMEMBER: In general, output indicators are counts while outcome indicators measure longer term changes in % or proportion in knowledge, attitudes, or practices ~20-30 minutes for group work ~30 minutes for group presentation/group critique You will have ~30 minutes for this activity, and if there is time, each group will share one logic model with added indicators with the larger group. I will let you know when 10 minutes remain. At the end of this activity, you should have one logic model written out on the large paper to be put on the wall that includes at least four indicators. As you are completing this activity, be sure to refer back to the Guide. [MONITOR THE GROUPS TO MAKE SURE THEY ARE PROGRESSING AND TO ANSWER QUESTIONS.] ACTIVITY RECOMMENDED MATERIALS FOR THIS ACTIVITY ARE: [YOU MAY PASS OUT BLANK PAPER FOR PARTNERS TO WORK ON OR YOU MAY PASS OUT ENOUGH COPIES OF THE POWERPOINT SLIDE “BLANK LOGIC MODEL_WITH INDICATORS” FROM THE TRAINING ACTIVITY FOLDER FOR GROUPS TO USE FOR THIS ACTIVITY] LARGE PIECE OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP THICK MARKING PENS, EACH GROUP SHOULD HAVE PENS FROM THE PREVIOUS ACTIVITY

Small Groups Activity: Adding Indicators to Your Program Logic Model
INPUT PROCESS OUTPUT OUTCOME IMPACT Indicators: [TEMPLATE FOR THE ACTIVITY IS IN THE POWERPOINT SLIDE “BLANK LOGIC MODEL_WITH INDICATORS” FROM THE TRAINING ACTIVITY FOLDER.] INPUT PROCESS OUTPUT OUTCOME IMPACT 127

Module 6: Indicator Matrixes for M&E
[MATERIALS FOR THE MODULE: 1) HANDOUTS OF THE SLIDES, AS NECESSARY; 2) COPIES FOR ALL PARTICIPANTS OF THE MEASURE EVALUATION PUBLICATION, “A GUIDE FOR MONITORING AND EVALUATING POPULATION-HEALTH-ENVIRONMENT PROGRAMS (The PDF of the guide is online here: 3) PRINTOUTS OF THE WORD DOCUMENT, “PHE COMPLETE INDICATOR MATRIX EXAMPLE” FROM THE TRAINING ACTIVITY FOLDER; 4) PRINTOUTS OF THE WORD DOCUMENT, “BLANK INDICATOR MATRIX EXAMPLE” FROM THE TRAINING ACTIVITY FOLDER; 5) PRINTOUTS OF THE POWERPOINT SLIDE “BLANK LOGIC MODEL_WITH INDICATORS” FROM THE TRAINING ACTIVITY FOLDER; 6) PRINTOUTS OF THE WORD DOCUMENT “INDICATOR REFERENCE SHEET_GUIDED” FROM THE TRAINING ACTIVITY FOLDER; 7) LARGE PIECES OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP, FOR ACTIVITY; 8) THICK MARKING PENS, ONE OR TWO PER GROUP.]

Type of Framework Brief Description Use for Program Management Basis for Monitoring and Evaluation of PHE Conceptual Shows the interactions of various factors that operate in your program environment Determine which factors the program will influence and how Helps explain and promote the multi-sector influences and integrated approach Logic model Logically links inputs, processes, outputs, and outcomes of planned activities Shows the causal relationship between inputs and reaching the objectives Shows intended progress of activities at all stages of the program – aids indicator selection Indicator matrix Logically links program goal, objectives, outputs, outcomes, and activities Shows the causal relationship and steps between activities and meeting objectives Guides monitoring and aids assessment by stating the information you will collect, how, when, where, and why. So far, we’ve talked about the conceptual framework and logic models. We will continue working on these frameworks – building on previous work to make a more valuable framework for M&E. Also, we have put our PHE problem into its greater societal or community context in the conceptual framework. Then, we created some logic models to help us show the logical connections between the inputs and processes and outputs of planned activities – helping link your activities to the program’s objectives (intended outcomes) and goals (longer term impact). Then, we added indicators to those logic models, bringing us closer to making our M&E plan and knowing how we will measure our success. Now, we are going to talk about the indicator matrix that we used to put it all together for M&E planning. The indicator matrix, often in spreadsheet or table format, lays out the programmatic steps that you will take to achieve your objectives, the ways that you will measure progress in both the short- and long-term, the methods you will use to collect this information, and the way that this information could be used to improve your program. By completing all three frameworks, you will have a guide to move from program planning, to implementation, and to final assessment. 129

Type of Framework Brief Description Use for Program Management Basis for Monitoring and Evaluation of PHE Conceptual Shows the interactions of various factors that operate in your program environment Determine which factors the program will influence and how Helps explain and promote the multi-sector influences and integrated approach Logic model Logically links inputs, processes, outputs, and outcomes of planned activities Shows the causal relationship between inputs and reaching the objectives Shows intended progress of activities at all stages of the program – aids indicator selection Indicator matrix Logically links program goal, objectives, outputs, outcomes, and activities Shows the causal relationship and steps between activities and meeting objectives Guides monitoring and aids assessment by stating the information you will collect, how, when, where, and why. [READ TEXT UNDER INDICATOR MATRIX] 130 130

Indicator Matrix Definition: a standardized summary of the project and its logic. Purposes: Summarizes what the project intends to do and how Summarizes outputs and outcomes that will be monitored and evaluated Details the indicators that will be used to measure progress and achievement of objectives Encourages discussion of data sources Indicator matrixes summarize what the project intends to do and how, the key assumptions, and the indicators of outputs and outcomes that will be monitored and evaluated. Indicator matrixes provide a streamlined perspective on the most critical processes contributing directly to program outputs and outcomes, and clarify the linear relationships between program decisions, activities, and products. Again, you may have seen similar frameworks called by alternative names. What is important is the components that you include and the thought process that you go through to write your framework. [IF YOU USE AN ALTERNATIVE, YOU MAY PRESENT IT HERE OR MODIFY THE EXAMPLE INDICATOR MATRIX INCLUDED FOR USE IN THIS MODULE.]

Filling In the Blanks: Data Needs for M&E
Before you begin the indicator matrix, it’s important to think about data! What data do you need? What data do you have? What are the gaps in data availability? How can you fill the gaps internally? How can you get the data externally? Before we can pull all of our knowledge together into writing the framework, we need to establish what data is actually required for our M&E. [READ SLIDE] Collecting data can take up time and resources, so it’s important we collect what we will use, and only what we will use. The bottom line is worth repeating: it’s important to gather the right data – and only that data – that you will actually use for decision making, monitoring, and evaluating your program!

How Do We Get Data for Decision Making?
Qualitative Methods Focus groups Key informant interviews Participatory rural appraisal/rapid appraisal techniques Quantitative Methods Adequacy and plausibility assessments Surveys Pre-post training tests To fill in the final framework, the indicator matrix, we need data! How can you get the information you need for your indicators? This is a topic that is too large and broad in scope for this training course, but here are some brief ideas on data collection ideas for you to consider. We will briefly discuss assessment techniques in the next module for how you can put these methods to use to gather data for M&E. [MORE INFORMATION ON THESE DATA COLLECTION TECHNIQUES CAN BE FOUND IN THE SUPPLEMENTAL PHE M&E RESOURCES FOLDER.]

Internal Data Sources Data that is generated by your own organization
Routine sources: procurement records, trip reports, budgets, facility records, training records Non-routine sources: surveys, participatory research findings I also wanted to briefly discuss some of the data issues, constraints, and special needs of PHE programs. Let’s talk for a minute about information that you gather for your programs – internal data. Commonly, this is used for routine monitoring, for example during your training activities. Or, it could be sporadic – such as a more extensive survey of households or conservation areas. The major point here is that you collect it, you pay for it, you have control over its quality, and you have control of its availability. [READ SLIDE]

External Data Sources Who else is working where you are working?
Conservation Partner District Health Officer Park Ranger Health NGO Community Council District Level Environment Officer Family Planning Services NGO Women’s Self-help Group Farmer’s Cooperative External data is from outside of your specific organization or project. In contrast to internal data, you cannot completely control the specific nature of this data, its timing, its contents, or its quality. But, its major advantage is that you do not have to collect it! For the PHE context, relying on external data is common as we frequently do not have the resources to gather all the data we need. Think about the data you need versus the data you want. Thinking about your data gap, what might your first step be to fill that data need? A first step to addressing gaps in data availability is to identify who else is working in the same region or area as you. These could be government or non-governmental organizations. It may be helpful to map them out like in this example. What kind of data might these different organizations collect? Some data will be collected routinely during day to day operations (e.g., records of contraceptive supplies ordered, records of contraceptive supplies distributed). Some data might be collected intermittently, during a program review or audit. Thinking about external data source may reveal linkages and opportunities for collaboration.

Data Source Issues for PHE: Sharing Data
If other organizations are gathering data your organization could also use, you could establish a data sharing partnership. Sharing data with other organizations reduces the burden of data collection on any one organization. Contributes to stronger relationships. What are the barriers to data sharing? Most of us in PHE work with diverse partners in various fields of expertise and at various levels of intervention. We also work in multiple sectors. In order to monitor and evaluate our programs effectively and at a reasonable price, sharing data is a good option. Much of our data comes from secondary sources – i.e., you do not collect it yourself but use other people’s data. Between our partner organizations, we should share data and also experiences – tips on pre-/post-tests, training tips, etc. See what data other people are using, how they collected it, and what gaps they might have. Collaboration might help you and your partners fill gaps.

Evaluating Secondary Data
Characteristics of quality data: Valid: accurate measure of a behavior, practice, or task Reliable: consistently measurable in the same way by different observers/promoters/supervisors Precise: clearly written, well-kept records Measurable: does the data answer your question? Timely: available at time intervals relevant and appropriate in terms of program goals and activities Programmatically important: linked to achieving the objectives that are needed for your monitoring, evaluation, or impact assessment Does this look familiar? It should! The characteristics for quality secondary data look similar to the characteristics of quality indicators. If possible, it is always good to look at what data already exists that you can use. However, secondary data should undergo some sort of evaluation. Think carefully through these issues for secondary data sources.

Additional Questions for Consideration
Although it is a good idea to use existing data to save time, energy, and costs, you must also consider: Who collected the original data? How complete is it? What is its intended purpose? Is it available? Is it sensitive? What does it cost to use in terms of time, $, travel, etc.? [READ SLIDE] Remember, don’t reinvent the wheel! Collaborate and cooperate where possible. Use existing data when feasible.

Giving Credit Data and information sources always need to be properly credited in any reports, publications, presentations, or even internal meetings. This adds credibility to your information and builds trust in data sharing. Your organization can ask that your data is also properly credited when other organizations use it. Don’t forget that building partnerships and sharing data is hard work, and all partners should be properly credited. [READ SLIDE]

Pulling It All Together…

INPUT Human and financial resources Demand for disease intervention in community PROCESS Train park guards in diarrheal disease and use of oral rehydration salts Train local community members in safe water storage Form natural resource management committees in 10 communities near park OUTPUT (ORS) packets distributed Community members trained in safe water storage NRM committees formed OUTCOME 1. Reduction in acute diarrhea cases among children 2. Increased safe water storage 3. Involvement of NRM committees in promotion of safe water IMPACT Reduced morbidity /mortality from diarrheal disease among children OUTPUT # of NRM committees formed in the 10 communities near the national park # or ORS packets distributed (health 3) Number of educational sessions held about safe water sources, storage, etc. (in project/target area) % of households with access to improved water source (health 7) Guide Integration indicators # 2, 3, 4, or 5 may work! OUTCOME % of households with access to an improved water source (health #7) % of communities in target area with community based (NRM) plans (Env’t 1) Oral rehydration therapy use rate (health 13) % of households storing drinking water safely (health 11) Value Added Indicators: Guide value-added indicators #1, 7,or 8 might work! Indicators: This is a past example that should look familiar. [CALL ATTENTION TO THE OUTPUT AND OUTCOME INDICATORS THAT ARE BOLD AND UNDERLINED.] IMPACT INPUT PROCESS 141

Section of PHE Indicator Matrix
Overall Objective 2: Reduce childhood morbidity in priority biodiversity areas in target communities Intermediate Result 2.1: Improved information about and access to safe drinking water Data Source Frequency Baseline End-of-Project Target Discussion Points/Comments Information User Outputs (what is the specific result of your activity?): Educational sessions held about safe water sources, storage, etc. (In project/target area.) Program Records Bi-annually This does not cover how many people attended, or if the people who attended were those that required training or whether they adopted the information. Who wants this information? Donors? Program managers? Outcomes (what change do you expect to see in knowledge, attitudes or practices as a result?): Percent of households storing drinking water safely (in project/target area). Household survey; household visits Annually Check available data 90% of house-holds The distribution of water storage units, and training to properly use the units, are listed as output indicators above. [AS THE FACILITATOR, YOU WILL NEED TO FAMILIARIZE YOURSELF WITH THE M&E PHE INDICATOR MATRIX BEFORE GETTING TO THIS SLIDE. THE SLIDE SHOWS A VERY BRIEF SEGMENT OF THE LARGER INDICATOR MATRIX, AND YOU WILL USE THE COMPLETE INDICATOR MATRIX TO ILLUSTRATE THESE POINTS TO THE GROUP. AFTER SHOWING THIS BRIEF SLIDE, WALK THEM THROUGH THE EXAMPLE INDICATOR MATRIX INCLUDED IN THIS TRAINING COURSE. HIGHLIGHT THE GOAL AND OBJECTIVES (WHICH WERE INTRODUCED IN PREVIOUS MODULES), AND WALK PARTICIPANTS THROUGH PIECES OF THE FULL EXAMPLE – HIGHLIGHTING HOW THE FRAMEWORK IS BROKEN DOWN INTO THE HEALTH, POPULATION, AND ENVIRONMENT COMPONENTS AS WELL AS A SECTION ON M&E. DRAW THEIR ATTENTION TO THE COMPONENTS OF THE EXAMPLE THAT ARE MOST APPLICABLE OR SIMILAR TO THE PROGRAM THEY ARE WORKING ON.] For M&E purposes, the logic model and its indicators are often reorganized to look like this, an indicator matrix. [AGAIN, IF YOU USE A DIFFERENT NAME OR ORGANIZE THIS MATRIX DIFFERENTLY, THIS IS A GOOD PLACE TO INTRODUCE YOUR OWN FRAMEWORK FOR PULLING IT ALL TOGETHER.] This may be a familiar framework for people working in health and population programs, but it may look new to people from other sectors. Regardless of the format, the information you need to think about is the same across all sectors. How you display it is not as important as knowing what you need and how to think logically through your indicators. The indicator matrix lists indicators by level just like we did in the logic model. In this framework, we concentrate on indicators at the outputs and outcomes level – the levels most likely to help you measure your progress and results within the PHE context. Impacts, not included, generally require more time and greater financial resources to show. In this table format, we add additional details like the data source, frequency of data collection, the level of the indicator at baseline, the intended end-of-project, discussion points, and who will want/need/use this information. The most pertinent comments from your M&E planning should also be listed in the framework. This slide shows two example indicators, each at a different level, and some additional information about these indicators. This same example is in the handout you received for this module, the example of PHE M&E indicator matrix. Notice the output and outcome indicator listed here were both included in the logic model activity we did before. They should look familiar to you! Let’s look through the table, feature by feature. First, the overall objective (from health, population, environment, or monitoring and evaluation) is shaded above each section of indicators. Also shaded is a box labeled, “intermediate results” (IR). Intermediate results are short term outcomes that are expected during the program and are used to measure progress towards overall objectives. [AGAIN, CHANGE THIS LANGUAGE IF NECESSARY. SOME MIGHT USE THE PHRASE, “INTERMEDIATE RESULTS”, “DEVELOPMENT RESULTS”, “INTERMEDIATE OUTCOMES”, ETC.] In this slide, intermediate result – 2.1 (improved information about and access to safe drinking water) – is an expected result that will lead to achievement of overall object #2, to reduce childhood morbidity in priority biodiversity areas. Under each intermediate result (2.1, for example), it lists key output and outcome indicators. The short term monitoring that can help you gage your progress (outputs) come first (as you would see this results first). Then, you can measure outcomes, longer term progress towards your goal/objective. Again, the language is not important; the methods and implementation are critical. [READ THROUGH SLIDE. ANSWER QUESTIONS. ASK PARTICIPANTS TO THINK ABOUT THE ANSWERS FOR THE INFORMATION USER COLUMN.]

Large Group Activity In one group we will:
Examine pieces of the example PHE indicator matrix Let’s think through: The four overall objective indicators – think about 2-5 years – do they work? Note that each objective is further divided into intermediate results. Under each objective sub-heading – population, health, conservation, and M&E – we will look at one outcome indicator (total of four outcomes) Under each objective sub-heading – population, health, conservation, and M&E – we will look at one output indicator (total of four outputs) [~ minutes] As a group, let’s pull out the sample M&E for PHE indicator matrix. Let’s look through it together as it will serve as a model for your next activity in your small groups. Let’s look at the front page first, looking at its organization, and reading through the goals and overall objectives. What do you think about the goal? Are the overall objectives SMART? Notice that there is an overall objective for health, child morbidity, conservation, and M&E – four objectives in all. Note that there are also four overall objective indicators – four longer term outcomes that will indicate overall progress. These could also be called “impact indicators” as they will be used at the end of the program to measure success. Note that each objective is further divided into intermediate results, or sub-objectives. Now, let’s look through the section marked, “Health.” It has a series of intermediate results working towards achieving the overall health objective. The intermediate results are further broken into sub-categories. This is a huge program; your program might be the size of only one objective or even one intermediate result. Under each health intermediate result (IR), there are outputs and outcomes – let’s look at a few. [SELECT 1-2 OUTCOMES, 1-2 OUTPUTS, 1-2 INTERMEDIATE RESULTS, ETC., THAT WORK BEST FOR YOUR GROUP OR ARE RELATED/PARALLEL TO YOUR SPECIFIC PROGRAM.] While we look through this, let’s discuss a few key points about this comprehensive example and move from this example to making your own indicator matrix. First, this is a comprehensive example, and very few, if any, programs would ever attempt to implement such a complex program. It is intended to provide you with many examples of outputs, outcomes, and intermediate results. It is not suggested that you replicate this model; rather, it is hoped that some of the rows in the indicator matrix might look familiar or work for you in your specific project. Again, keep in mind that you would typically need one output or one outcome for each key activity. You do not need to have a 1-to-1 correspondence for every output and outcome to every intermediate result. Although this example is not representative of a typical indicator matrix for a single organization or program, it could be an example of a joint framework of several integrated organizations that partner on this multi-sector approach. With that in mind, you could look at this framework also as a representation of how organizations could partner, share data, share resources, and work together from program design, planning, implementation, and M&E activities. The data source could list which organization was responsible for the collection of the indicator with additional comments about sharing that data with others. Lastly, and perhaps most importantly, if you are having trouble thinking through each column in the matrix for a specific indicator, consider reevaluating the quality of that indicator or its utility for your program. For example, if the last column in the information used for a specific indicator in your indicator matrix is blank or you have difficulty thinking about who wants/needs or would use the information from the indicator, it is likely that you should NOT collect that indicator. Remember: collect only the information you, your donors, or your stakeholders need for program performance and assessment! Collect indicators that are timely, programmatically oriented, feasible, etc. Does everyone feel comfortable with the organization of the indicator matrix? [REVIEW A CONSERVATION OR OTHER AREA OF THE SAMPLE FRAMEWORK AS NEEDED/DESIRED. ASK FOR QUESTIONS. NOTE: IF YOU DO NOT WRITE INDICATOR MATRIXES THE EXACT SAME WAY, FEEL FREE TO MAKE CHANGES OR SUGGEST CHANGES TO THE PARTICIPANTS BEFORE THE NEXT ACTIVITY.]

Preparing to Create Your Indicator Matrix
Questions for you to discuss as you plan your indicator matrix: What data is available to you? What data do you need? What could you collect? What would you like if it were available? What sources of data could you explore? How many “standard indicators” can you include? Now, let’s return to the logic models you made before about the activities in your specific programs. You’ve already added indicators too, so you are 75% there! Now, we will put that information into this format, adding in the critical details. In your small groups, before you begin, you should consider these questions. Return to your notes on indicators as you work through your own indicator matrix. [READ SLIDE] [THESE QUESTIONS ARE BEST USED IF THEY ARE ON THE BOARD DURING THE LARGE GROUP, PARTNER, AND SMALL GROUP ACTIVITY OR AS A HAND OUT.]

Small Group Projects Use the PHE programs M&E indicator matrix as an example. Using your own project/program, you will: Use your two outputs and two outcome indicators from the previous activity on adding indicators to logic models. Are the indicators P,H, E, I, or V? Write your own indicator matrix, including filling in across all rows, for these four indicators (follow the format of the indicator matrix example). For one indicator, complete the indicator reference sheet. Present your findings to the group. [~ 1-2 HOURS, ADJUST AS NEEDED. THIS IS CRITICAL!] For this activity, return to your small groups from the conceptual framework and logic model activities. Using your own projects, the conceptual model you created, and the logic models you finished in earlier modules, you will now create an indicator matrix for your own project like the one we just reviewed. Your indicator matrix will not be as extensive as the one we just looked at: remember, that one was for illustrative and learning purposes. For this activity, you will further develop the indicators you already listed as part of “adding indicators to logic models.” Using the example indicator matrix we just looked at as an example, you will write your own indicator matrix with the four indicators you developed in previous activities – two outputs and two outcomes. Then write an indicator reference sheet for one of the indicators. [FOR THOSE PARTICIPANTS WHO ARE ADVANCED OR WISH TO SPEND MORE TIME ON THIS CRITICAL ACTIVITY, YOU COULD ADD ADDITIONAL COMPONENTS TO THIS ACTIVITY. YOU COULD HAVE PARTICIPANTS DEVELOP OUTPUT AND OUTCOME INDICTORS FOR ACTIVITIES IN THE P, H, AND E SECTORS. FOR EXAMPLE, YOU COULD HAVE THEM CONSIDER ONE P, H, AND E ACTIVITY FROM THEIR PROGRAM. THEN, THEY COULD DEVELOP ONE OUTPUT AND ONE OUTCOME INDICATOR FOR EACH SECTOR – FOR A TOTAL OF SIX INDICTORS. THEN, ALSO FROM THEIR PROGRAM, YOU COULD HAVE PARTICIPANTS DEFINE AN OUTPUT AND AN OUTCOME INDICATOR FOR AN INTEGRATED ACTIVITY AS WELL AS A VALUE ADDED ACTIVITY. BE CREATIVE HERE AND ADAPT THIS ACTIVITY TO CHALLENGE YOUR PARTICIPANTS. USE THE EXAMPLE PHE INDICATOR MATRIX; HAVE THEM COMPLETE AS MUCH AS WOULD MEET THEIR NEEDS.]

Module 7: Assessment and Evaluation Designs for PHE
[MATERIALS FOR THE MODULE: 1) HANDOUTS OF THE SLIDES, AS NECESSARY; 2) COPIES FOR ALL PARTICIPANTS OF THE MEASURE EVALUATION PUBLICATION, “A GUIDE FOR MONITORING AND EVALUATING POPULATION-HEALTH-ENVIRONMENT PROGRAMS (The PDF of the guide is online here: 3) PRINTOUTS OF THE WORD DOCUMENT, “PHE COMPLETE INDICATOR MATRIX EXAMPLE” FROM THE TRAINING ACTIVITY FOLDER; 4) PRINTOUTS OF THE WORD DOCUMENT, “BLANK INDICATOR MATRIX EXAMPLE” FROM THE TRAINING ACTIVITY FOLDER; 5) LARGE PIECES OF POSTER BOARD OR FLIP CHART PAPER, ONE PER GROUP, FOR ACTIVITY; 6) THICK MARKING PENS, ONE OR TWO PER GROUP.]

Objectives of the Session
By the end of this session, participants will be able to: Define what is an adequacy assessment and an evaluation study Describe what assessments and evaluation studies are most appropriate for low resource PHE programs Understand how to select the best design for their specific study How can we tell if our program has made a difference, whether we have accomplished our objectives and whether an intervention works? For smaller programs, like most PHE programs, it is often sufficient to know if changes have occurred without needing to know whether these changes occurred solely due to a particular intervention. In other words, we are not always interested in making causal inferences. For example, if the local council wants to know whether improved agricultural techniques are being used in certain villages, a study could be conducted to determine this without trying to attribute these changes to any particular intervention. Studies that help determine the level of outcomes of interest, without determining causal linkages with specific prior events or interventions, are known as non-experimental or observational studies. These types of studies are important and are the most frequently-implemented ones. Hence, it is important to understand their strengths and limitations. For most PHE programs, we will use these designs. [READ SLIDE] In this session, we will learn about differences between two types of non-experimental studies and how to make these designs work for programs in lower resource settings. It is beyond the scope of this module and this course to provide instruction on how to complete more complicated evaluation or impact studies. However, in the reference section of this training program are some additional materials. Also, the MEASURE Evaluation MENTOR course provides intermediate level instruction on these more rigorous designs.

Capacity and Resources for M&E
Capacity for M&E M&E unit or dedicated M&E person The M&E person should know if the survey is done well, even if not doing the survey in house Scale of program funding and resources for M&E Ten percent of resources devoted to M&E Costs related to data collection systems & information dissemination Costs may be higher during the first year Other helpful resources include indicator guides, manuals, communication tools Before we start, we need to consider the resources necessary to carry out M&E activities. Each organization or program should have someone dedicated to M&E. If you do not have someone specifically trained and dedicated to M&E, it is most important that the M&E person or team knows how M&E is effectively undertaken even if they do not have the “in-house” abilities to do all the work themselves. Remember that at least 5%, but better 10%, of your program/project resources should be designated for M&E. This may sound like a lot, but consider how important it is to not only implement the program, but to show how money was spent, who used the services, whether a difference was made, and whether that difference may be due to your program. This budget should include funding for proper reporting forms, data management systems, and plans to use the data for decision making. Keep in mind that the first year funds may be greater than in the middle years as you will need to gather baseline data and create all of your M&E systems; one time items that last the life of a program. Also, keep in mind that you should budget money for studies at the end of the program to determine if changes were made over the course of the program.

Answer what kind of questions?
Type of assessment Answer what kind of questions? Adequacy Did the expected changes occur? Are objectives being met? Were activities performed as planned? Does not require a comparison group. May benefit from a pre-/post-test design. Plausibility Did the program seem to have an effect above and beyond the contribution of other, non- program influences? Requires comparison groups. There are two types of assessments that are more appropriate for the financial and technical resource level of most PHE programs. There is plenty of information on the MEASURE Evaluation website on how to conduct more formal evaluations, but we are not going to consider them today. For today, we are going to think about the kind of inference (adequacy, plausibility) that we need to make and the questions we need to answer. We will also consider what our stakeholders and what the decision-makers need to know. In general, these types of assessments answer these key questions [READ SLIDE]. Overall, an adequacy assessment answers the question: Did the expected changes occur? This is most common for PHE programs and the least expensive. A plausibility assessment wants to find out whether the program seems to have an effect above and beyond other external influences. For some of us, we will be able to implement these types of assessments with additional technical and financial resources. You should note that these types of assessments require both an intervention and a comparison group that does not receive the intervention.

Adequacy Assessment Adequacy studies describe whether an objective (outcome) is met or not. Typically addresses provision, utilization, or coverage aspects. No need for a comparison group. Question answered: Are expected levels achieved? Can also answer questions of evaluation (amount of change) if you do pre- and post-tests. Question answered: Is the difference between program and non-program communities equal or greater than expected? Most of us in PHE will conduct adequacy studies or an adequacy assessment. Although this is not the strongest evaluation design, it is certainly helpful to demonstrate program progress and meeting objectives. It can be very helpful for both internal information needs and for reporting requirements. However, because this is a weaker design, it must be done well or it is not effective in measuring results. [READ SLIDE] These studies will help you show what services you provided, who used the services, and where the program was utilized. This type of study will help you determine if you provided the intended program, who you serviced, and where they accessed the program. To make this type of study better, you could add a pre-test to know what level of service provision, knowledge, attitudes, or practices exist BEFORE the program begins. If you do a pre-test, you can answer different questions and know, to a greater degree, whether some of the changes you observe are due to your program versus changes from another program. Why does adding a pre-test improve your assessment? [POSSIBLE ANSWERS: IT TELLS YOU THE LEVEL OF THE OUTPUT/OUTCOME OF INTEREST IN THE TARGET POPULATION BEFORE YOU BEGAN SO THAT YOU CAN DETERMINE THE AMOUNT OF IMPROVEMENT OVER THE COURSE OF YOUR PROGRAM WHEN YOU DO YOUR POST-TEST AT THE END. IT ALSO HELPS YOU PLAN YOUR PROGRAM BETTER IF YOU ARE ABLE TO SET A TARGET BASED ON THE INITIAL SITUATION IN THE COMMUNITY. IN THIS CASE, USING THE SAME TOOL FOR PRE-TEST AND FOR POST-TEST IS VERY IMPORTANT FOR CONSISTENCY AND RELIABILITY OF INFORMATION.]

Features of Adequacy Assessment
Pros: If only input or output (activity) results are needed, this may be a good design for you. Simplest (and cheapest) of evaluation models. Data are needed only for outcomes (think about your logic framework!). Cons: If there is no change, you will not know whether the lack of change is due to program failure or not. You cannot use this design to measure impact – you will not know if changes are due to the program alone. This type of design has several advantages and disadvantages. [READ PROS] First, this is a fine design if you are most interested in reporting on your activities and the immediate results of these activities, i.e., people trained, people who participated in your campaign, etc. This is also a simple and relatively inexpensive design since you only collect data once and only at the end of the program. You could also use this design to look at your outcomes, or the intermediate changes that result from your program. However, these types of designs have some distinct disadvantages. [READ CONS] First, this type of design will not let you know if the changes you observe are due to you or to other, external, changes. Think about this: if you only collect data at the end of the program from users of your program, how would you know if a change happened? What was the community like before? Maybe the results would have been the same before the program? Or, if you see an improvement, did your program make the change? Do you know all the other programs and changes happening in the community at the same time? Second, considering that you cannot know if changes are due to your program, you cannot use this design for impact. Impact evaluation requires a comparison group, which is not a component of this design.

Adequacy assessments designs
One program group – post-test only (weakest design) One group, pre- and post-test (better) IMPLEMENT PROGRAM ASSESS TARGET GROUP AFTER PROGRAM ASSESS TARGET GROUP BEFORE PROGRAM ASSESS TARGET GROUP AFTER PROGRAM Let’s look at two examples of adequacy assessments. In the first design, there is only one group – the group/area/community where you are doing your intervention. This might just be a pilot program. In this type of adequacy assessment, you would administer a post-test (e.g., survey, focus groups, post-test, community mapping) to your target group once, only after participants received an intervention. What do you think are the weaknesses of this design? [PROBE: HOW DOES YOUR GROUP COMPARE TO OTHER PEOPLE? WHAT WAS THE LEVEL OF YOUR TARGET GROUP BEFORE YOU STARTED YOUR PROGRAM?] This design is common and relatively inexpensive, but it does not allow you to measure and compare changes from baseline (before the intervention), nor does it allow you to measure change in relation to other groups of people who did not take part in the intervention. Notice there is no pre-test so you don’t know the level of the target group before you start. And, there is no comparison group, so you don’t know if your target group is like other people or not. The second design here is better. Why do you think this design is better? [PROBE: DO YOU KNOW WHAT THE LEVEL OF THE TARGET POPULATION WAS BEFORE THE PROGRAM BEGAN? DO YOU KNOW IF YOUR TARGET POPULATION IS LIKE OTHER POPULATIONS?] In the second design, you assess your target group both before and after program implementation. The strength of this design is that it provides baseline information that you can compare with your post-intervention data. For this design to work, you must administer the same instrument (same focus group questions, same interview questions, same questionnaire, same community mapping technique, etc.) in the same way both before and after the program. This design can tell you whether your target group made improvements, but cannot assure you that your program was responsible for the outcomes. Why not? [PROBE: WITH WHOM DO YOU COMPARE THE PRE- AND POST-TEST?] Alternative explanations are still possible (e.g., change occurred because participants changed over time). Although these are weak assessment techniques, they are flexible. Sometimes, with limited funds and resources, these are the only options that we have for evaluating our PHE programs, especially for pilot programs. However, if you use one of these designs, and do your survey/focus groups/training tests, etc., well, you could use the post-test data as your pre-test data for an add-on, scaled-up, or improved program. Or, you could do multiple post-program tests and see results over time. You should not let resources or a weaker design prevent you from doing the best you can do. Keep in mind that every baseline and end line assessment has potential to be used more than once or in other programs. Lastly, when planning assessment, consider your partners and their needs as well. Combining talents, technical skill and resources is an excellent way to improve your M&E. Maybe two organizations could create a joint questionnaire or share the funds necessary to implement a baseline assessment even if you do separate assessments post program. IMPLEMENT PROGRAM

Plausibility Assessment: Post-test Only 3.
BEFORE PROGRAM AFTER PROGRAM ASSESS TARGET GROUP AFTER PROGRAM IMPLEMENT PROGRAM Community 1 ASSESS COMPARISON GROUP AFTER PROGRAM Community 2 Now, let’s look at how we could make improvements to the adequacy assessment and strengthen our ability to determine whether the changes we see are due to our program or to other factors. This is the third design, a plausibility study. What do you think makes this a plausibility study versus an adequacy assessment? What improvements have we made in this design over the first two designs we just discussed? [PROBE: DO YOU ASSESS BEFORE AND AFTER THE PROGRAM? IS THERE A COMPARISON GROUP? DO YOU DO A PRE-TEST IN BOTH INTERVENTION AND COMPARISON GROUP? DO YOU DO A POST-TEST IN BOTH?] In the third design, you assess both your target group and another similar group that does not receive the program before and after implementation. The addition of a comparison group helps you determine whether or not your target group would have improved over time even if they had not experienced your program. The more similar the two groups are with respect to variables that may affect program outcomes (e.g., gender, race or ethnicity, socioeconomic status, and education), the more confident you can be that your program contributed to any detected changes. [PROBE: HOW CAN YOU HELP IMPROVE THE SIMILARITIES BETWEEN YOUR INTERVENTION GROUP AND YOUR CONTROL? ARE THERE WAYS THAT YOU CAN THINK OF THAT WOULD STRENGTHEN THE COMPARISON BETWEEN GROUPS?] Again, your assessment could be focus groups, participatory rural appraisal technique, survey questionnaire, etc. Whatever assessment method you choose, use the same method twice for more comparable results. NO PROGRAM!

Plausibility Assessment: Post-test Only
Identify people/communities who received the program. Identify similar people/communities who did not receive the program. Test both groups USING THE SAME QUESTIONNAIRE OR FOCUS GROUP OR EVALUATION METHOD. Compare results between program and non-program groups. This is better than an adequacy assessment because of the comparison group. Let’s review the steps needed to complete the previous study: [READ SLIDE] Example: A community based family planning distributor wants to see if outreach-education about the advantages of family planning will increase the number of women who seek contraceptives from her distribution booth. She asks another family planning distributor, from a very similar community, to help her. The women compared their distribution records, and about the same number of women used contraceptives in both communities. Then, the first distributor went out into the community and actively promoted contraceptives through household visits, community meetings, and just chatting to women. The second distributor did nothing new – no outreach. After three months, focus groups were held in both communities with similar women of reproductive age. The focus groups were conducted with the same questions in both communities – centering around a discussion of contraceptives, side effects, advantages, etc. The women in the community of distributor #1 seemed much more positive about family planning methods. The second group of women were not as favorable. New users of contraceptives also went up in the first community. The plausibility design suggests that outreach works and that women are responding positively to the outreach activities.

Plausibility assessment: pre- and post-test 4.
BEFORE PROGRAM AFTER PROGRAM ASSESS TARGET GROUP BEFORE PROGRAM ASSESS TARGET GROUP AFTER PROGRAM IMPLEMENT PROGRAM ASSESS COMPARISON GROUP BEFORE PROGRAM ASSESS COMPARISON GROUP AFTER PROGRAM This is the fourth design – an improvement over the previous three. What makes this design different? Why might these changes be beneficial for determining whether your program achieved its objectives and goal? [PROBE: WHEN DO YOU ASSESS THE PROGRAM GROUP? THE COMPARISON GROUP? WHO DO YOU TEST IN THE POST-TEST?] In the fourth design, you assess both your target group and another similar group that does not receive the program before and after implementation. The addition of a comparison group helps you determine whether or not your target group would have improved over time even if they had not experienced your program. The more similar the two groups are with respect to variables that may affect program outcomes (e.g., gender, race or ethnicity, socioeconomic status, and education), the more confident you can be that your program contributed to any detected changes. This design also helps control for test effects (e.g., improvements on the post-test due to participants' experience with the pre-test). Let’s consider the previous example with the community based distributors. How could we make that example fit this assessment design? How would our conclusions be stronger? [ASK THEM: COULD YOU ADD A PRE-TEST (AN INITIAL FOCUS GROUP) IN BOTH COMMUNITIES TO DETERMINE BASELINE ATTITUDE ABOUT FAMILY PLANNING AMONG THOSE WOMEN? HOW WOULD HAVING THE BASELINE IMPROVE THE DESIGN? WHAT ADDITIONAL QUESTIONS COULD YOU ANSWER? WOULD YOU KNOW WHETHER THE DIFFERENCES IN AFTER PROGRAM POST-TEST KNOWLEDGE WAS DUE TO YOUR PROGRAM? HOW COULD HAVING THE BASELINE STRENGTHEN YOUR CLAIM THAT KNOWLEDGE IMPROVED MORE IN THE PROGRAM COMMUNITIES?]. Can someone give me an additional example using a survey or mapping exercise? Overall, using this design, you can make a case that the changes you see are due to your program. If you measure the knowledge, attitude, practice at baseline in similar communities, implement the program in only one of the communities, and measure the same information using the same measurement tool (survey, focus groups, etc.), you’ll have data for a plausibility assessment. Keep in mind that you might need to consult an evaluator or seek technical assistance to do this design, and the other plausibility design, correctly. NO PROGRAM!

Plausibility Assessment: Pre- and Post-test
Identify people/communities who get the program. Identify similar people/communities who do not get the program. Pre-test both groups USING THE SAME QUESTIONNAIRE OR FOCUS GROUP OR METHOD. Deliver the intervention/program only to the program group. Post-test both groups with the same method after the program finishes. Compare results between program and non-program group; consider assistance from a skilled evaluator or consultant. How do we go about this type of study? [READ SLIDE] This fourth design offers the greatest opportunity to attribute changes in outcomes to program activities by adding a comparison group to your pre- and post-program design. To accurately analyze this information, you should consider hiring or seeking technical assistance.

Choosing the Right Design for You
Adequacy designs will allow you to link program outcomes to program activities. If you want to have more certainty, try to use a pre- and post-test design. If you want to be more confident, you should try to use a comparison group – it will be more costly. If you have the resources, hire an evaluator to help determine which design will maximize your program’s resources and answer your team’s evaluation questions with the greatest degree of certainty. Disseminate your findings and share lessons learned! Overall, let’s review what we just learned. [READ SLIDE] There are many resources to help you create questionnaires and conduct analysis – but this is beyond the scope of this short training. Many of these resources are presented at the end of the training materials or on the MEASURE Evaluation website.

Areas of Concern to Different Decision-Makers
Type of Evaluation Provision Utilization Coverage Impact Adequacy PHE Programs Health center manager International Agencies District health managers Plausibility Donor agencies Scientists Probability Donor Agencies & Scientists Let’s not forget that it is important for us to think of our stakeholder and funder needs. This slide shows areas which are typically of interest to different decision makers. Health center managers usually want to know about the provision and utilization of services to help them improve service delivery, while district health managers and international agencies are usually more interested in knowing whether a certain level of coverage (program reach) was met. For the type of decisions made by many PHE program managers and health clinic managers, adequacy evaluations are often sufficient to provide necessary assurance that the expected goals are being met. International agencies and donors are often interested in whether the changes made are likely caused by the program versus other external causes, and they may be more likely to require plausibility assessments of coverage or impact to justify further investments in a program. Some donor agencies have been trained to regard probability assessments as the gold standard. However, other donor agencies and scientists are most likely to require probability evaluations – the ones we did not cover during this session as most of the time probability evaluations are not feasible for the routine evaluation of program effectiveness. Source: Habicht, Victora and Vaughan (1999).

Summary Assessment Objective What It Says Data Needs Adequacy:
(Assessment of change in outcome.) Assess whether impact was reached. Indicates whether resources were well spent or not. Outcome data collected among participants. Baseline data improves study design. Plausibility: (Uses comparison group.) Understand what affects the outcomes. Helps understand how and why the program succeeded or failed. Outcome data collected among participants and comparisons. Baseline data improves study design. Before moving on to the activity and putting our knowledge into practice, let’s review once more. [Read slide] [START A SMALL DISCUSSION AS A REVIEW AND CLARIFICATION. ASK:] What type of adequacy assessment or evaluation design might work for your program? What steps can you take to improve your assessment design now?

Small Group Activity – Assessment Planning
Take the four indicators you detailed in your indicator matrix (you may consider more indicators if you choose); Review the data source for your output and outcome indicators; Now, considering your data needs, your actual budget, and your program stage, discuss an assessment strategy: What type of assessment could you do (of four designs)? Can you afford a baseline? Comparison group? Is it too late? How will you collect the information for your indicators? (Household survey, facility survey, household visits, program records, focus groups, interviews, community mapping, etc.)? What specific information do you need? (Think about numerator and denominator.) What questions will you ask? Who will you ask? What forms might you need for monitoring outputs? What skills/knowledge will your assessment team need to look at outcomes? [PARTICIPANTS WHO SUCCESSFULLY COMPLETE ALL COURSE MODULES AND HAVE AN INTERMEDIATE KNOWLEDGE OF M&E OR PROGRAM PLANNING SHOULD BE ABLE TO COMPLETE THIS ACTIVITY. PREVIOUS FAMILIARITY WITH DATA COLLECTION AND ASSESSMENT (FOR BOTH THE FACILITATOR AND PARTICIPANTS) WOULD BE HELPFUL. YOU MIGHT NEED TO ADJUST THE CONCEPTS YOU WISH PARTICIPANTS TO DISCUSS BASED ON THEIR LEVEL AND YOUR ASSESSMENT OF THEIR ABILITIES. YOU COULD ALSO DIRECT PARTICIPANTS TO SOME OF THE ADDITIONAL RESOURCES ON PARTICIPATORY RURAL APPRAISAL, QUALITATIVE METHODS, SURVEY DESIGN, ETC., INCLUDED IN THE MATERIALS BEFORE STARTING IN THIS FINAL ACTIVITY TO BRING IT ALL TOGETHER.] For the next 1-2 hours, you will work in your small groups to consider what type of assessment design you could use for your specific program. You’ll need to think about measuring both outputs and outcomes. You’ll need to think about setting up your study from the beginning or doing the best you can as a mid-course assessment or final assessment if your program has already started. This is a difficult, but invaluable task. Look at these five sets of questions as you work in your small groups. You will be able to think through many of these answers, but other questions will only generate more questions. That’s okay. I am most interested in your thorough discussion, your attention to the details, and your ability to think creatively through solutions. There are no right answers, so just try your best. After 1-2 hours, each small group will present. There will be time for feedback, suggestions, comments, and questions. Feel free to use your group presentation as an opportunity to seek advice from your peers or to present specific challenges that others in the group may help you overcome. This is a learning opportunity! [GIVE THEM BETWEEN 1-2 HOURS DEPENDING ON THE LEVEL OF DETAIL YOU WISH THE PARTICIPANTS TO PROVIDE. THIS EXERCISE IS A STRETCH FOR SOME, BUT VERY VALUABLE FOR THOSE ABLE TO CONSIDER THIS LEVEL OF DETAIL.] [HERE IS ANOTHER, SIMPLER, IDEA FOR A FINAL DISCUSSION ON INDICATORS AND ASSESSMENTS: LOOKING BACK ON YOUR INDICATOR MATRIX FOR THE FOUR INDICATORS YOUR WORKED THROUGH. FIRST, REVIEW THE DATA NEEDS YOU IDENTIFIED AND THE POSSIBLE SOURCES OF DATA, KEEPING IN MIND THE ISSUES THAT YOU PRESENTED IN THE LAST MODULE. THEN, CONSIDER: WHAT TYPE OF BASELINE DATA YOU WILL NEED AND HOW TO COLLECT IT. IF YOU ALREADY STARTED YOUR PROGRAM, CONSIDER WHETHER YOU CAN COMPLETE A MID-POINT EVALUATION OR AN END OF PROGRAM SURVEY. WHERE CAN YOU GET SECONDARY DATA? WHAT ALREADY EXISTS THAT YOU CAN USE? WHAT ASSESSMENT DESIGN MIGHT WORK FOR YOU?]

Small Group Activity – Presentations
One person from each group should present your major discussion points 10 minutes per group In your brief presentation, include information on: Indicators you are using Possible assessment design to collect the data (numerator/denominator) Data collection type (surveys, focus groups, etc.) Anticipated difficulties Expected successes [DEPENDING ON THE LEVEL OF THE PARTICIPANTS, THEY MIGHT NEED HELP WITH THIS ACTIVITY, INCLUDING CLARIFYING YOUR EXPECTATIONS. IF YOU HAVE LOCAL FACILITATORS, THIS WOULD BE A GOOD TIME TO HAVE ONE FACILITATOR WITH EACH GROUP TO HELP GUIDE DISCUSSION. ALTHOUGH THE ACTIVITY COULD BE A STRETCH FOR SOME, MANY SHOULD FIND THIS EXERCISE A GOOD WAY FOR THEM TO BRING THEIR M&E DISCUSSION AND TRAINING FULL CIRCLE WITH AN OPPORTUNITY TO BRING THE PIECES OF THE TRAINING TOGETHER.] [AFTER 1-2 HOURS, PULL THE GROUPS BACK TOGETHER FOR THESE BRIEF PRESENTATIONS. ALLOW SOME FLEXIBLE TIME FOR OVERFLOW IF GROUPS SEEM HIGHLY MOTIVATED IN THEIR DISCUSSIONS. AFTER EACH SMALL GROUP PRESENTS, ALLOW TIME FOR FEEDBACK, SUGGESTIONS, COMMENTS, AND QUESTIONS. MAKE SURE THAT ALL GROUPS RECEIVE BOTH POSITIVE AND NEGATIVE FEEDBACK – OR, MAKE SURE TO HIGHLIGHT POSITIVE THINGS ABOUT EACH GROUPS’ EFFORT. REMIND PARTICIPANTS THAT LEARNING AND COOPERATING WITH PEERS IS AN IMPORTANT PART OF M&E.]

Conclusion, Wrap-up, and Evaluation
[THIS IS WHERE YOU WILL COMPLETE A POST-TEST EVALUATION OF THE TRAINING. AN EXAMPLE IS PROVIDED IN THE COURSE MATERIALS OR YOU CAN DEVELOP YOUR OWN. THIS IS WHERE YOU WILL DISCUSS THE PRESENTATIONS, TAKE FINAL QUESTIONS, AND PLAN FOR NEXT STEPS.]

Training Modules: Monitoring and Evaluation of Population, Health and Environment (PHE) Programs MEASURE Evaluation [COMMENTS, NOTES, AND INSTRUCTIONS.

Similar presentations

Presentation on theme: "Training Modules: Monitoring and Evaluation of Population, Health and Environment (PHE) Programs MEASURE Evaluation [COMMENTS, NOTES, AND INSTRUCTIONS."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Training Modules: Monitoring and Evaluation of Population, Health and Environment (PHE) Programs MEASURE Evaluation [COMMENTS, NOTES, AND INSTRUCTIONS.

Similar presentations

Presentation on theme: "Training Modules: Monitoring and Evaluation of Population, Health and Environment (PHE) Programs MEASURE Evaluation [COMMENTS, NOTES, AND INSTRUCTIONS."— Presentation transcript:

Similar presentations

About project

Feedback