1. Evaluation Methods

2. Objectives
Describe the most common quantitative and qualitative methods used to evaluate ACSM.
Demonstrate the role of these methods in supporting ACSM evaluation.
We just discussed the five main categories of evaluation. Now we will talk about the specific methods we use to actually conduct evaluation. These methods can be either quantitative or qualitative.
Review objectives.

3. Evaluation of Treatment Support Program
Quantitative
Qualitative
Patient data from
10 TB clinics.
_____________________________
How many patients left treatment early.
How many weeks of treatment were completed.
% default by gender, ethnicity, age.
Interviews with 20 patients who left treatment early.
_____________________________
What they experienced during treatment.
Why they left treatment.
Ideas for improving the treatment support system.
Let’s distinguish the differences between these two kinds of methods with an example. Let’s consider a social mobilization objective of training former TB patients to be treatment supporters. This slide summarizes the different kinds of data we could use to better understand the success of this objective.
Review slide.
What are the main differences you see? (One side numbers; other side expressed in words, ideas.)
Quantitative methods are useful for helping us summarize data across a population and get estimates. Qualitative methods help give us in-depth understanding of behavior. For example, analysis of TB patient data may tell us, on average, how many left treatment early or even if there was a correlation between how long they stayed in treatment and the number of contacts they had with their treatment supporter. A qualitative method would help us understand why these patients left treatment early and give us more detail about how the treatment supporter program is working.

4. What are Quantitative Methods?
Analyze numerical data.
Use standard tools (e.g., questionnaires).
Need precision and reliability!
Often use large sample sizes with sophisticated methods to select participants.
Results are generalized to a population.
Example: The TB prevalence survey planned for Nigeria will include 49,000 people selected from 700 clusters representing six zones.
Quantitative methods give us the numerical data to estimate indicators for an entire population. For example, a TB prevalence survey will tell us approximately what percentage of a country’s population has smear-positive TB at a certain point in time.
We generally use quantitative methods when we need numbers (how much, how many, how often, etc.), or need to measure our performance against some other benchmark or target.

5. What are Qualitative Methods?
Provide in-depth, detailed information on behavior, opinions, ideas, etc.
Analyze text, words, pictures, or interview transcripts instead of numbers.
Use open-ended data collection tools.
Usually smaller sample sizes.
In general, quantitative methods produce “hard numbers,” while qualitative methods capture more descriptive data. Qualitative data include words (or pictures) rather than numbers. With qualitative methods, there is less emphasis on counting numbers of people who think or behave in certain ways and more emphasis on explaining why people think and behave in certain ways. They want to show nuances and context behind the numbers.
Qualitative methods usually involve smaller numbers of respondents. Therefore, you cannot generalize findings to a larger population.
We generally rely on qualitative methods when we need to know why something is happening or to get opinions about a topic.

6. Quantitative Methods Analysis of routine surveillance and NTP data.
Analysis of program or project data.
Simple surveys.
Population-based surveys:
KAP, prevalence.
Let’s start by discussing these common quantitative methods that can be used to support ACSM evaluations, from baseline assessment to impact evaluation. There are many others, but these are the ones you are most likely to use given the amount of resources you probably have for evaluation.

7. Method: Analysis of Routine Surveillance and NTP Data
Existing Ministry of Health reporting system data:
Vital registration data.
Routine reports from disease-specific programs.
Hospital data.
Data usually available; no special data collection needed.
Quality may be a concern: completeness, timeliness, accuracy may not be good.
A lot of details are already available to us through existing surveillance and disease reporting systems. These data can give us information about patient demographics, diagnostic and treatment statistics, and how services are used, how frequently, and where.
But at this broad level, we must be cautious about data quality. At the regional and national levels, data come from many sources over continuous periods of time, so data can often be incomplete, delayed, or inaccurately reported. So we must scrutinize surveillance data very carefully.

8. Examples: Routine Surveillance
Number of deaths due to TB in State B in 2010.
Prevalence of TB/HIV co-infection by age and gender.
Number of TB cases reported in 2012.
Percentage of MDR-TB cases among new reported cases in 2012.
These are some examples of routine public health surveillance data.
Review slide.

9. Using NTP Data to Prioritize ACSM
Quarterly review of SS+ case notification
Which districts report an increase in SS+ case notification?
Which district reported the highest number of retreatment cases?
Where is TB/HIV co-infection the highest?
Here is a very simple example of how routine surveillance data could be used in evaluation. You could collect the indicator sputum smear-positive case notification at baseline and then analyze that quarter to look for trends, such as:
Which districts are reporting increases in SS+ case notification?
Which district has the highest number of retreatment cases?
Where is TB/HIV co-infection the highest?
If you see trends, particularly in gaps or problems, you could use ACSM to address those gaps. For example, if a district is steadily reporting more MDR-TB cases each quarter, you might use that information to prioritize social mobilization to support patients through treatment in that district.
What category of evaluation would this be?
Encourage responses. (Baseline to prioritize ACSM activities, then outcome to determine the contribution of those activities.)

10. Method: Analysis of ACSM Program Data
Collect and analyze key output and outcome indicators over the life of the project.
Data come directly from your project.
Quantitative indicators and sources:
Advocacy.
Communication.
Social mobilization.
Another important quantitative method is analysis of our own ACSM program data. These are the output and outcome indicators that you collect on a regular basis to determine if you are implementing the project as planned and achieving anticipated results. The data come directly from your project and are included in your M&E plan (including the detailed indicator descriptions and guidance on how to analyze them).
At a basic technical level, this would be considered monitoring. But for many ACSM programs, this may be the only realistic source of quantitative data they have. But we will see in a minute how monitoring data can become useful in evaluation.

11. Analysis
Expected Outcomes
Activity
Community health workers identify and refer persons with presumptive TB.
Increased number of people arriving at DOTS center for screening
% of referrals received at DOTS center
Increased number of TB cases diagnosed
% of referred persons diagnosed with TB
Increased number starting treatment
% of TB cases starting treatment
Let’s assume that case notification is low in our district. We decide to partner with and train existing community health workers to identify people with possible TB symptoms and refer them to the local DOTS program.
As a result of this activity, we hope to see more people going to the clinic for screening, more TB cases diagnosed, and of course, more TB patients starting on treatment. We can collect data on these different indicators to do some simple analysis.

12. TB Screening Referrals from Community Health Workers, District X, 2011
57%
55%
40%
Let’s assume our community health workers keep track of their referrals using referral slips. The DOTS centers are also keeping track of the referrals that come in, so we can look at the percentage of people who received referrals who actually came to the DOTS center (percentages listed at top).
Point to corresponding bars and percentages.
Then we can see the percentage of those people who were diagnosed with TB, and then how many started treatment. Almost all TB cases diagnosed are starting treatment, so we have left out that percentage.
How do you think the program is doing? What does the analysis tell you?
Encourage a brief discussion.
These numbers are nice for telling us the outcomes for the specific people our program referred. But we cannot see how our referrals fit in with ALL those screened in the district. So we need a larger data set beyond our own program. 8%
18%
20%
15%

13. Contribution of Community Health Workers to TB Case Notification in District X, 2011
Quarter
Number of TB cases diagnosed after referral
Total number of TB cases notified in district
Contribution of referral network to case notification
Jan – Mar 8 60
13%
Apr – Jun 9 73
12%
Jul – Sept 12 85
14%
Oct – Dec 10 93
11%
Therefore, we need to link our data to the larger context of overall case notification trend so we can better understand how well our intervention is working or contributing.
This final piece of analysis combines our program data with the case notification data. Here, we can determine the contribution of the community health workers to case notification in the district. We use the number of TB cases diagnosed among those who were referred to DOTS and went to the clinic and the total number of TB cases reported during the same time period to estimate the overall contribution of our intervention to case notification.
What story does this tell about our program? How is this data analysis more useful than just program data alone?
Encourage a brief discussion.

14. Give Your Data Meaning and Importance
Interpret results within a larger context.
Link project data with routine/NTP data to show contribution.
Monitoring: How many did we refer and how many were diagnosed with TB?
Evaluation: How much did our referrals ultimately contribute to overall case notification in the district?
Interpreting the values and trends of our project within the context of what is happening in the community and overall TB indicators gives our data more meaning and importance. This further illustrates the difference between monitoring and evaluation.
Read examples.
Monitoring shows that we did some work. It then becomes evaluation when we determine if that work had any larger outcomes. In essence, does our work really contribute to anything?
This is why frameworks are so important, because that is how we show the connection between our activities and the broader NTP objectives or context (in this case, increased case notification).

15. Method: Population-Based Surveys
Provide data on priority indicators that can be generalized to a population.
% of adults 15 years and older who correctly identify chronic cough as a TB symptom.
Usually requires a large research team, specialized expertise, and a full-time survey manager.
Nigeria 2011/2012 prevalence survey includes 49,000!!!
Population-based surveys are another quantitative method that we can use to evaluate our ACSM activities. They are very large-scale efforts to survey a large group of people. They usually require a large team with specialized expertise and a full-time survey manager.
Review examples on slide.
Remember:
Defining your population is very critical: Do you want to know something about the general population? About people at highest risk for TB? About DOTS nurses? It is important to be very specific about your target so you can design an appropriate sampling strategy to get representative data.
These surveys are a huge effort! It is important to weigh the costs of using this method with the benefits (e.g., the data) you will get out of it. Always ask if there is another way you could get the same data. And think carefully about how you will use the data. Nothing is worse than collecting lots of data, writing up a report about it, and then not using it to inform your program or make decisions.
KAP surveys and prevalence surveys are very important population-based surveys for TB and for ACSM.

16. What is a KAP Survey? KAP = knowledge, attitudes, and practices
Population-based survey—very large sample size!
Face-to-face interviews with standard tool.
Provides summary data on KAP.
Averages, percentages, totals, ranges, etc.
A KAP survey is a population-based survey, so this means that the information we get can be generalized to a population. The population could be any group, such as all adults aged 15 and older, all DOTS nurses in your country or region, all TB suspects visiting clinics in a district, etc.
We use face-to-face interviews with a standard tool in a KAP survey. We do not record the answers with audio tape (as we might with a focus group). We ask everyone the same questions and then record their answers on a questionnaire in a standardized way.

17. What does a KAP survey tell us?
Knowledge, attitudes, and practices
What does the population know about TB?
What myths or misinformation are common?
What stigmas are related to TB?
Where do people seek care?
What media channels are used by this population?
KAP stands for “knowledge, attitudes, and practices.” KAP surveys provide summary data about a large population. The results include averages, percentages, totals, ranges, etc. We will have lots of numbers to look at after a KAP survey.
A KAP survey can answer many questions about a population’s knowledge, attitudes, and practices.

18. Essential Components: KAP Survey
Appropriate sampling strategy
Standardized, pre-tested questionnaire
Field protocol and manual
Trained, UNBIASED data collection team
Data analysis plan
Data use strategy
BUDGET!
These are the basic, minimum elements of a KAP survey.
Review components.
It is really difficult to design and implement a KAP survey effectively. And as you can see, it requires a lot of financial and human resources. So if you do not have experience in this area or sufficient funding, you will need to seek an outside expert or rely on another quantitative method to get the information you need.

19. Method: Simple Surveys
Information from a smaller, targeted sub-group:
Village surveys.
Basic questionnaires.
Polls.
Faster and easier, but less scientifically rigorous than KAP surveys.
Cannot generalize the results to a larger population.
You are more likely to use a method called a “simple” survey. These are surveys or questionnaires distributed to a much smaller, more targeted group of people. Examples might include an in-person oral survey of tribal elders, an electronic questionnaire sent to ACSM coordinators across your country, or an informal poll taken among random participants attending a World TB Day event.
These types of opinion or community surveys are a good choice if you need some numbers related to a specific sub-group of people. They are quick and less expensive to conduct than a full KAP survey. But because the sample size is small and your sampling methods are not scientifically rigorous, you cannot reliably apply your results to a broader population. The results can only be a “best guess” as to how a larger, more diverse group might respond.

20. Qualitative Methods Focus group discussions Key informant interviews
Exit interviews
Media scans
These are the qualitative methods you are most likely to use to support ACSM evaluations, from baseline assessment to impact evaluation. We will describe each one in more detail.

21. Method: Focus Group Discussions
Small group (6–10 people).
Gather opinions, insights, thoughts, and feelings about a topic.
Led by a moderator; assisted by a notetaker.
Semi-structured guide with key questions.
May be recorded, transcribed, formally analyzed with software.
Useful for pre-testing communication materials.
A focus group discussion is a semi-structured conversation with a small group of people who share something in common. They are usually members or sub-groups of the population your intervention is targeting. The group is led by a moderator (facilitator) who follows a question guide designed to get specific information or opinions from the group.
Focus group discussions are particularly useful for pre-testing communication materials or gathering formative information to help you design a project. They are useful anytime you need to collect as many diverse opinions as possible on a specific subject within a short period of time.
Refer to Handout 4.2 Guide to Effective Focus Group Discussions for more information.

22. Method: Key Informant Interviews
Face-to-face interview with critical individuals:
Policymakers, opinion leaders.
Those who represent key stakeholders (e.g., NGO directors, association leaders).
Marginalized, vulnerable populations who do not want to participate in groups.
Good for collecting sensitive, personal, or confidential information.
Limited perspectives.
This method is very useful when you need to collect detailed information on opinions/insights from someone (usually a leader/key stakeholder) who may not want to participate in a larger group or would prefer to remain anonymous. The person being interviewed may be more honest when s/he knows the information is confidential.
Many times, it is also a practical issue. A busy policymaker or leader may not have time to participate in a focus group discussion, and this can be a good way to get information from him/her quickly.
One key limitation is that you need to choose informants carefully so you get a range of opinions, especially if you are using this method to design an advocacy strategy. You need to understand all sides of an issue, so do not leave out key stakeholders that could influence your activities.
Can you think of some examples of using key informant interviews? Who would you interview and why?
Solicit responses and briefly discuss.

23. Interviews or Focus Groups?
What do patients think of a new brochure on treatment adherence?
Focus Group
How can we get the directors of TB clinics and HIV centers to collaborate more?
Interviews
What do HIV-positive injection drugs users know about TB symptoms and diagnosis?
Focus Group or Interviews
THIS SLIDE IS ANIMATED.
Even though interviews and focus groups can get us similar information, usually one method is better than the other based on the goal and participants. I will show you an example of an evaluation question and you decide which method would be better to answer the question and why.
CLICK and read the first example. Ask one table for its opinion.
CLICK to reveal answer.
A focus group would be better here because we want as many opinions as possible and the subject matter is not too personal.
CLICK and read the next example. Ask another table for its opinion.
Here, interviews would be better because it is probably impossible to schedule several directors for the same meeting. And they may not be candid with their opinions in front of their peers.
CLICK and read the last example. Ask another table for its opinion.
In this example, you might be able to use either. Because of the sensitive nature of injection drug use and HIV confidentiality, you may need to use interviews. However, what if these patients attend an HIV support group together or we use peer referral to find participants? You might be able to do a small focus group in these cases. But interviews are the more likely choice.

24. Exit Interviews
Evaluate short-term outcomes of changes to service delivery.
Standardized instrument used to gather data on patient experiences with a clinic or specific provider.
Can measure patient satisfaction and monitor provider behavior after training.
Exit interviews are borrowed from the private sector. With this method, a trained interviewer uses a standard instrument to interview patients or clients as they leave a facility to capture what each patient just experienced.
One example is exit interviews of patients leaving a local DOTS center, to see whether or not a training was effective in changing HIV counseling and testing practices among nurses. Patients could be asked:
What message did you hear from nurses?
Did you feel welcomed?
How were HIV testing procedures explained?
The responses can be used to assess patient satisfaction and to improve how HIV testing is offered and conducted for TB patients.

25. Media Scans Search media for how a specific topic is covered:
What are the most common messages about TB?
How often is TB discussed in the media? Is it a “hot” topic?
Is the information about TB accurate?
Are the messages stigmatizing?
Useful to evaluate interventions with journalists.
Formal or informal.
Media scans involve searching for material related to a specific topic (e.g., TB) over a specific time period to determine how often it is covered, what types of messages are being communicated, the accuracy of the information, and how stigmatizing it may be.
It can be a useful method when you are considering working with the media to improve their communication about TB.
You can scan print media (newspapers, magazines) as well as radio and television. Within some contexts, it is increasingly important to look at Internet and social media if these are key sources of information about the topic.
A formal media scan hires a clipping service or research firm to search specific newspapers, magazines, etc., for articles that they then systematically analyze using a scoring system. They score articles on levels of accuracy and whether or not the articles provide appropriate, non-stigmatizing information.
Staff members can conduct informal media scans by looking over the key media outlets on a routine basis to quickly identify coverage of the issue.

26. Q&A/Resources Refer to Handout 4.3 More Qualitative Methods.
There are three other qualitative methods you may have heard about or have used in the past: direct observation, rapid assessment, and case studies. These usually require more time, resources, and staff training, so they may not be a good first choice for smaller programs.
There are two other helpful resources. The first is the Guide to Developing Knowledge, Attitude, and Practice Surveys, published by the Stop TB Partnership, which is available on their website. The second is this publication from Family Health International (now FHI 360), which is a helpful field guide to using qualitative methods.
Are there any questions about the methods we have discussed so far?