1. Indicators for ACSM

2. Objectives
Explain the role of indicators in monitoring and evaluation for ACSM.
Describe the characteristics of well-defined indicators.
Demonstrate how to develop indicator descriptions.
In this session, we will explain what indicators are, what makes a good indicator, and how to develop effective indicator descriptions.

3. Creating a Framework
Let’s take another look at our framework for our case example.
Ask: How will we know we have these outputs? Solicit ideas and discuss.
The easiest way is to count them. But how meaningful will that simple number be? How can we add some context to that? We could compare it to a target or baseline.
Ask: How will we measure these outcomes? What will indicate that we have achieved these? What will our specific evidence be? Solicit ideas and discuss.
Now let’s look at impact. What evidence could we examine to prove this impact?
Solicit ideas and discuss.
We just created what are called “indicators” for this framework.

4. What Is an Indicator? Clues, signs, or markers. Used to:
Track inputs, activities, outputs, and outcomes.
Measure progress toward the goal and objectives.
Crow and the Pitcher!
Indicators are the signs or markers that we watch for to “indicate” what is happening. We use indicators to keep track of our planned work and the results of that work.
When we collect and analyze the correct indicators over time, we know that (1) we carried out the work we said we would do, and (2) whether or not the work achieved the objectives it was intended to support.
Think back to our story about the Crow and the Pitcher. Can you remember some of the indicators from that story? How could the crow measure what he did and the effect afterward?
Encourage responses. (Counting # of pebbles, level of water, amount of thirst.)

5. Indicators are Part of the M&E Plan
Framework
Indicators
Data Collection
Data Quality
Data Use and Reporting
Evaluation
Strategy
Budget
Indicator descriptions are another part of your M&E plan that we are building this week. Your monitoring and evaluation efforts will be designed to help you collect these indicators. Without indicators, you will not know what data you will need as evidence of your ACSM success.

6. Examples Framework Component Indicator
INPUT: IPCC training (curriculum)
100 copies of training manuals, slide sets, and handouts on IPCC
ACTIVITY: Lobbying
Meeting with the finance minister and NTP director
OUTPUT: Patient coalition meeting
Number of patients attending the meeting
OUTCOME: Increased funding
Percentage of NTP budget covered by the Ministry of Health
OUTCOME: Increased support for community-based DOTS
* Policy change to allow volunteer health workers to serve as treatment supporters
OUTCOME: Increased political support for TB control
* Parliament declares TB a national emergency
Here are some examples of indicators for different components of a framework. You can see that they offer more detail to define that component, which helps everyone understand them in the same way.
In this first example, we use an indicator to define the INPUT of curriculum.
Read INPUT example.
In example #2, we have an indicator to define our ACTIVITY of “lobbying.”
Read ACTIVITY example.
Indicators can also define OUTPUTS.
Read OUTPUT example.
Finally, we have some indicators that explain some OUTCOMES. Read OUTCOME examples.
Notice that indicators can be quantitative (in black) and expressed numerically as counts, percentages, rates, etc. They can also be qualitative (in green with asterisk). They are expressed as a description (something you witness or observe but cannot count or calculate with numbers).

7. Indicators: Important M&E Evidence
Goal
What am I trying to achieve?
Objective
What are the major steps I need to take to reach my goal?
Activities
What am I going to do to reach my objective?
Indicators MONITOR progress: Have I done what I said I would do?
Inputs
What resources do I need to complete each activity?
Outputs
What will the immediate product of my activities be?
Outcomes
What do I hope will happen as a result of my activities?
Indicators EVALUATE results: Did my work have the desired effect and contribute to my goal?
Impact
What do I think these activities will contribute to my goal?
As we said before, indicators help you answer important monitoring and evaluation questions. Activities, inputs, and outputs relate to monitoring. So indicators for these components will help you answer the question, “Have I done what I said I would do?”
Outcomes and impact require evaluation. So your indicators will answer the question, “Did my work have the desired effect and contribute to my goal?”
Review slide in more detail.
Refer to Handout 3.1 Common ACSM Indicators.
This handout offers a list of some useful output and outcome indicators for common advocacy, communication, and social mobilization activities. This is not a complete list, but it may be a helpful place to get some ideas.
Briefly review Handout 3.1 Common ACSM Indicators.

8. Outputs versus Output Indicators
Output is the immediate result of an activity:
# of people trained
# of people with TB symptoms going to health facility for evaluation
REPORT WHAT YOU ACTUALLY ACCOMPLISHED
Indicators for outputs measure your result against a target value:
# of people actually trained versus how many you wanted to train
# of people with TB symptoms going for evaluation versus how many you thought would go
ASSESS WHETHER YOU REACHED YOUR TARGET VALUE
Many people get confused when it comes to indicators for outputs. At first it would seem that an output indicator might be the same as the output itself. You will know how many people you trained by simply counting the number of people you trained. But there is a subtle difference between an output and its indicator.
An output is simply what you can count right after the activity is completed. But that number alone does not tell you very much. There is no indication if this is a good number or not unless we compare it to our target. When we compare the output with its target, we have a better idea if we were successful in producing that output. That comparison becomes our output indicator.
Review slide in more detail.

9. Inputs Activity Outputs Outcome Impact
NTP Goal: Reduce morbidity and mortality due to
TB in Country X.
Indicator:
Number of TB deaths per 100,000 per year
NTP Objective: Increase case detection rate from 42% to 60% by 2015.
Indicator:
Case detection rate
ASCM Objective: Raise knowledge of TB symptoms and TB services to increase the number of people in City X seeking care for TB symptoms at DOTS centers by 30% by December
Indicator:
# of people requesting screening at the City X DOTS center compared to baseline
Inputs
Funding
List of subway routes
Activity
Develop and produce subway ads
Outputs
# ads produced
# subway trains with
ads
Outcome
Increased knowledge of TB and DOTS centers
Impact
Increased # of TB cases detected
THIS SLIDE IS ANIMATED.
Let’s look at another framework for a communication objective.
Starting with NTP Goal, review each component of the framework in order. Then CLICK to display the indicator after it.
Indicators:
% of people who know two TB symptoms and have heard of the DOTS center
# of people requesting screening at City X DOTS center
Indicator:
Difference in case detection rate between intervention and control cities
Indicator:
Lists developed
(Yes/No)
Indicator:
Number of ads and subway trains versus target
Indicator:
Ads produced according to schedule (Yes/No)

10. Main Steps of Creating Indicators
Select an indicator
Test against criteria
Write a description
Now let’s talk about the process of creating effective indicators. There are three main steps. First, we select an indicator we think might work. Then we test that indicator against some basic criteria that we will talk about next. Then we revise that indicator with a more complete description.

11. Characteristics of Good Indicators
These are the characteristics of good indicators.
Briefly review each characteristic.
Once you have selected an indicator, ask yourself these questions to see if your indicator “passes the test.” If you answer no to any of these questions, you need to revise your indicator.

12. Is This Indicator Valid?
Does it tell us what we really want to know?
Could it actually measure something else?
Example:
% of Ministry of Health budget dedicated to TB as an indicator of government commitment to TB control.
Improved indicator:
% of NTP budget covered by government (compared to % covered by donors).
THIS SLIDE IS ANIMATED.
In this case, valid means correct or true. Does the indicator tell us what we really need to know? Could the indicator measure something else?
In this example, we want to monitor the budget dedicated to TB over time as an indicator of the government’s commitment to TB control.
What are some potential problems with this indicator? (Depends on the denominator. What if the Ministry of Health budget changes a lot over time? What if the NTP has the money it needs, regardless of its percentage of the Ministry of Health budget? What if all of that money comes from external donors? How could you improve this indicator?)
Discuss possible problems and revisions.
CLICK to display new indicator.
This new indicator will tell us over time if the government steps up and provides more of the budget for TB. It is more valid because it will not measure something else, like changes in the overall Ministry of Health budget or in donor contributions.

13. Is This Indicator Reliable?
Will everyone interpret or calculate it the same way?
Example: Number of partners actively participating in advocacy coalition.
Improved indicator: Number of partners who attend at least 75% of coalition meetings.
Example: Smear conversion rate.
THIS SLIDE IS ANIMATED.
A reliable indicator is one that everyone knows how to measure or calculate in exactly the same way.
Is this indicator clear to everyone?
Discuss elements that are not clear. (What does it mean to “actively” participate? Just come to some meetings? How many? Lead or join a committee?)
How could we make this indicator more clear?
CLICK to display new indicator.
CLICK to display second example. Our indicator in this example is “smear conversion rate.”
How reliable is this indicator?
Discuss possible problems and revisions. (No clear definition. What is the denominator?)
To make sure we have reliable indicators, we first need very clear definitions. Then we need to train our staff on those definitions.

14. Is This Indicator Activity Specific?
Does it tell us about our activity only? Could any other factor (ACSM or not) influence this indicator?
Example: % of TB screening clients receiving a smear test as an indicator that providers from our training are referring TB suspects properly.
Improved indicator: % of TB screening clients with documented referral for smear microscopy.
THIS SLIDE IS ANIMATED.
How specific is this indicator to our activity? Could anything else increase or decrease the number of smear tests (e.g., other program is incentivizing testing; our providers are referring, but the laboratories have chronic stockouts of supplies).
In one country, the NTP invested a lot in IPCC training for DOTS nurses, including a review of referral practices and an emphasis on providing patients with the information they need in order to obtain a smear test and a referral from the clinic to the laboratory. The first indicator they thought of using was percentage of TB suspects receiving a smear test. But within this context, the laboratories were in terrible shape. In reality, the DOTS nurses were doing exactly what the NTP wanted, but TB suspects did not get the smear test because the laboratories did not have the supplies they needed. So we had to think of a better indicator.
CLICK to display new indicator.
We must always consider what other factors besides ACSM could affect our indicators. This is where our gap analysis of all of the individual-, group-, and system-level barriers is really helpful.

15. Is This Indicator Feasible?
Do we have a realistic data source? Do we have enough money and staff?
Example: % of population with correct knowledge about TB symptoms, annually (indicator of a successful communication activity).
Alternative indicator: Number of people with possible TB symptoms presenting for diagnosis at specified DOTS clinics.
THIS SLIDE IS ANIMATED.
Feasibility may be the most important factor in our choice of indicators. Here is an example. If we are investing significantly in communication efforts to improve knowledge about TB in communities (with the ultimate goal of improving case detection), it would be nice to know the percentage of the population with correct knowledge about TB on an annual basis. But can we afford to collect it this often? What do you think?
Discuss elements that are not feasible. (Can we really afford a KAP survey every year? Do we have expertise for population-based surveys?)
Knowledge, attitudes, and practices, or KAP, surveys are very expensive, take a long time, and require specific expertise. Perhaps there is another result of our communication efforts that may be easier to collect and report.
CLICK to display new indicator.

16. Comparable
Do the results mean the same thing in different geographic areas at different times?
Example: Number of nongovernmental organizations in each region mobilized to participate in World TB Day.
Alternative indicator: % of nongovernmental organizations in each region mobilized to participate in World TB Day.
THIS SLIDE IS ANIMATED.
When indicators are comparable, they represent the same result or outcome in different places over time. It is important that we compare apples to apples and not apples to oranges.
Do you think this indicator is comparable across regions? What could be different across regions for this indicator? (Total number of NGOs in the region [influences denominator], weather of region in March, physical size of region.)
CLICK to display new indicator.

17. Indicator Descriptions
What is the complete definition?
What is the data source for this indicator?
How do we calculate the value of the indicator?
Numerator, denominator
Qualitative criteria
THIS SLIDE IS ANIMATED.
After we select the right indicators, we have to determine how we will define, collect, analyze, and report them. These are called indicator descriptions. With indicator descriptions, everyone understands our activities and results to mean the same thing.
These are the basic elements of an indicator description.
CLICK to display each bullet and briefly explain.
DEFINITION: Every part is clearly and fully explained.
SOURCE: All indicators need a data source. Specify the form, database, reports, lists, etc., where the data should be found.
CALCULATE VALUE: For numerical indicators, such as percentages, specify the numerator, denominator, and calculation. If you have simple counts, make sure it is clear who gets included in your count and who is excluded. For a qualitative indicator (e.g., policy change or political commitment), it is necessary to describe criteria for reporting the values.
Keep in mind, we may not be able to select the perfect indicator. We have to make choices, and sometimes our choice may be influenced more by what is practical and easiest than what is the most accurate or reliable indicator.

18. Indicator: Percentage of district DOTS nurses receiving IPCC training
Indicator Description
Indicator: Percentage of district DOTS nurses receiving IPCC training
Definition
Data Source
Calculation
Percentage of DOTS nurses who attend the full training and receive a completion certificate.
IPCC training attendance sheet and certificate list submitted to NTP.
Number of nurses completing the training.
Total number of DOTS nurses in the district.
This is a full description for this indicator.
Review slide in detail.

19. Questions?
Next, we are going to practice writing an indicator description for our case example. But before that, are there any questions? Where do you feel confused?