Session 5: Selecting and Operationalizing Indicators

Session Overview  Selecting indicators  Common difficulties with identifying indicators  Operationalizing indicators  Tools to use to operationalize indicators

Session Learning Objectives By the end of the session, the participant will be able to:  understand the process for selecting indicators;  be aware of the common difficulties with identifying indicators; and  understand how to operationalize indicators.

Selecting Indicators

Steps in Selecting Indicators Step 1: Review the program framework Identify what needs to be measured. Good indicators start with good program framework. Start with the overall goals and objectives for the program.

Step 2: Develop a list of possible indicators Identify indicators for each level of results. You can use:  indicators from past years of the program  experience of similar programs  global/regional/national indicators  indicator guides Steps in Selecting Indicators

Step 3: Assess each indicator Each indicator should be:  measurable  practical  reliable  relevant  useful for decision-making  precise  sensitive  capable of being disaggregated Steps in Selecting Indicators

Step 4: Select the “b est ” indicators  Narrow your list to the final indicators.  Aim for an optimum set that meets management needs at a reasonable cost.  Limit the number of indicators for each objective/result to 2-3.  Remember your target audiences, both external and internal. Steps in Selecting Indicators

Common Difficulties with Selecting Indicators

Common Pitfalls in Indicator Selection  Indicators not linked to program activities.  Using outputs as outcomes.  Poorly defined indicators.  Data needed for indicator is unavailable.  Too many indicators!!!

Pitfalls with Selecting Indicators Indicator not linked to program activities IR: Expanded access to diarrhea treatment services Activities: Train providers in treating acute diarrhea Inappropriate Indicator: Percent of facilities with adequate conditions to provide care Better indicators: Number of clinicians trained, # of facilities with a trained provider, percent of clinicians with 100% on post-test The program is not aiming to affect facility conditions, only provider skills.

Pitfalls with Selecting Indicators Using outputs to measure outcomes Problem: routine monitoring data (outputs) are available from households but outcomes are needed for reporting. You collect: Number of farms which practice recommended biosecurity measures You need: Percent of farms that practice recommended biosecurity measures Routine monitoring data should not be reported as outcome data. Compile your routine data as well as information about your target area to turn outputs into outcome measures.

Pitfalls with Selecting Indicators Indicator poorly defined Activity: A radio campaign to provide information about AI prevention methods Inappropriate indicator: Percent population with AI knowledge Better indicator: Percent population that can cite top 3 ways to protect their families from AI “Knowledge of AI” is vague. What knowledge is critical for the prevention of transmission of AI?

Pitfalls with Selecting Indicators Data needed for indicator not available Data issue: Information on stock-outs may not be collected daily Inappropriate indicator: Percent of days per quarter that service delivery points have stock-out of drugs Better indicator: Percent of service delivery points that had a stock out of drugs at some time during the last quarter If relying on routine data, indicator definition will depend on how data are collected.

Pitfalls with Selecting Indicators Too many indicators!!! Rules of thumb: one or two indicators per key activity or result (ideally, from different data sources) not more than 8-10 indicators per programmatic area at least one indicator for every core activity (e.g. infection control, laboratory improvement, outbreak response) use a mix of different data collection strategies and sources There is no set formula for how many indicators to use. Consider your budget, stakeholders, and program plans. Then consider: Is it reasonable? Feasible?

Adding Indicators to the Program Framework

Adding Indicators to Program Framework Input Activity/ Process OutputOutcomesImpact Quantifiable resources going in to your activities – the things you budget for. 1. What you do to accomplish your objectives? Immediate results from your activity – people trained, services provided Longer-term change in knowledge, attitude, behavior, etc. related to program goal Long-term, population level change. Can relate to a program or organization’s vision/mission statement Indicators (example) How do you measure this?

Logical Framework: Training Activity 18 INPUTPROCESSOUTPUT OUTCOME IMPACT Time and skills to develop new biosecurity training curriculum for market owners Conduct training events Market owners trained in biosecurity methods Improvemen t in market conditions Reduction in HPAI/H5N 1 virus circulation

Adding Indicators Input Activity/ Process OutputOutcomesImpact Time and funding to develop new biosecurity training curriculum for market owners Conduct training events Market owners training in biosecurity methods Improvement in market conditions Reduction in HPAI/H5N1 virus circulation in targeted areas Indicators (example) Amount of funding for curriculum development Percent of staff for curriculum development Number of trainings conducted Percent of market owners trained in biosecurity methods Percent of markets in which proper biosecurity methods have been implemented Percent of poultry in targeted markets found to be infected with H5N1

Small Group Activity Return to small groups and return to the logical frameworks from Session 3. As in the examples, fill in the indicators for your logical framework:  You must have indicators for every column.  REMEMBER: In general, output indicators are counts while outcome indicators measure longer term changes in percentages or proportions in knowledge, attitudes, or practices. After group work, share logic models with the larger group. You have about 30 minutes for this activity.

Small Group Projects Present your findings to the group.

Operationalizing Indicators

What does operationalizing indicators mean? Identify exactly how a given concept, result, or behavior will be measured. Challenges include:  subjective judgment  local conditions  unclear yardsticks

Importance of Operationalizing Indicators Operationalizing indicators helps ensure that:  everyone is using the same definitions;  units of measurement are the same;  everyone understands the indicators;  only the most significant program elements or achievements are being tracked; and  the chosen indicators will assist in appropriate decision- making and/or action-planning.

Exercise: Operationalizing Indicators In groups of 3-4 people, review and discuss how these indicators were operationalized :  Percent of provinces and districts with at least 5 or more surveillance officers trained on disease surveillance.  Number of poultry disease reports to central level per year.  Percent of provinces that have an avian influenza and pandemic preparedness plan.  Number (percent) of markets with active surveillance at least twice per month.  In case of outbreak, percent of outbreaks responded by rapid response teams within 24 hours.  Percent of provinces with an effective (staff-trained and PPE provided) rapid response team.  Percent of provinces with effectiveness of trained rapid response team tested through simulation exercise or outbreak.  Percent of AI specimens tested within 24 hours.  Percent of targeted influenza specimens tested.

Indicator Reference Sheet The sheet provides detailed documentation for each indicator:  basic information  description  plans for data acquisition  data quality issues  plans for data analysis, reporting, and review

Example of an Indicator Reference Sheet