Presentation is loading. Please wait.

Presentation is loading. Please wait.

0 Developing An Evaluation Plan For TB Control Programs Developing An Evaluation Plan For TB Control Programs Division of Tuberculosis Elimination National.

Similar presentations


Presentation on theme: "0 Developing An Evaluation Plan For TB Control Programs Developing An Evaluation Plan For TB Control Programs Division of Tuberculosis Elimination National."— Presentation transcript:

1 0 Developing An Evaluation Plan For TB Control Programs Developing An Evaluation Plan For TB Control Programs Division of Tuberculosis Elimination National Center for HIV, STD, and TB Prevention Centers for Disease Control and Prevention

2 1 Developing An Evaluation Plan For TB Control Programs Developing An Evaluation Plan For TB Control Programs Reference: A Guide to Developing an Evaluation Plan

3 2 Why Develop an Evaluation Plan? Provides a cohesive approach to conducting evaluation and using the results Provides a cohesive approach to conducting evaluation and using the results Guides evaluation activities Guides evaluation activities Explains what, when, how, why, who Explains what, when, how, why, who Documents the evaluation process for all stakeholders Documents the evaluation process for all stakeholders Ensures implementation fidelity Ensures implementation fidelity

4 3 Guide to Developing An Evaluation Plan Document referenced throughout presentation Document referenced throughout presentation Provides a template and instructions to help TB program staff develop an evaluation plan Provides a template and instructions to help TB program staff develop an evaluation plan Steps to evaluation are explained in detail Steps to evaluation are explained in detail Completing sections and tables will result in an evaluation plan Completing sections and tables will result in an evaluation plan

5 4 The CDC Program Evaluation Framework

6 5 Systematic method for evaluation Systematic method for evaluation Based on research and experience Based on research and experience Flexible and adaptable Flexible and adaptable Promotes a participatory approach Promotes a participatory approach Focuses on using evaluation findings Focuses on using evaluation findings

7 6 Sections of an Evaluation Plan Introduction Introduction Stakeholder Assessment Stakeholder Assessment Step 1: Engage Stakeholders Step 1: Engage Stakeholders Background and Description of the TB Program and Program Logic Model Background and Description of the TB Program and Program Logic Model Step 2: Describe the Program Step 2: Describe the Program Focus of the Evaluation Focus of the Evaluation Step 3: Focus the Evaluation Design Step 3: Focus the Evaluation Design

8 7 Sections of an Evaluation Plan Gathering Credible Evidence: Data Collection Gathering Credible Evidence: Data Collection Step 4: Gather Credible Evidence Step 4: Gather Credible Evidence Justifying Conclusions: Analysis and Interpretation Justifying Conclusions: Analysis and Interpretation Step 5: Justify Conclusions Step 5: Justify Conclusions Ensuring Use and Sharing Lessons Learned: Reporting and Dissemination Ensuring Use and Sharing Lessons Learned: Reporting and Dissemination Step 6: Ensure Use and Share Lessons Learned Step 6: Ensure Use and Share Lessons Learned

9 8 Introduction An introduction provides background information, identifies the purpose of the evaluation, and provides a roadmap of the plan. An introduction provides background information, identifies the purpose of the evaluation, and provides a roadmap of the plan. Evaluation Goal Evaluation Goal What is the purpose of the evaluation? What is the purpose of the evaluation? Evaluation Team Evaluation Team Who is your evaluation coordinator? Who is your evaluation coordinator? Who are the members of your evaluation team? Who are the members of your evaluation team? Reference: Table 1 in the Evaluation Plan Guide

10 9 Stakeholder Assessment Stakeholders are individuals with vested interests in the success of the TB program. Involving stakeholders increases the credibility of the evaluation and ensures that findings are used as intended. Who are the stakeholders in your TB program? Who are the stakeholders in your TB program? What are their interests in the evaluation? What are their interests in the evaluation? What role do they play in the evaluation? What role do they play in the evaluation? How do you plan to engage the stakeholders? How do you plan to engage the stakeholders? Reference: Table 2 in the Evaluation Plan Guide

11 10 Background and Description of the TB Program The program description ensures that stakeholders have a shared understanding of the program and identifies any unfounded assumptions and gaps. Need Need What problem does your program address? What problem does your program address? What are the causes and consequences of the problem? What are the causes and consequences of the problem? What is the magnitude of the problem? What is the magnitude of the problem? What changes or trends impact the problem? What changes or trends impact the problem?

12 11 Background and Description Context Context What are environmental factors that affect your program? What are environmental factors that affect your program? Target Population Target Population Does your program target the TB concerns of one population? Does your program target the TB concerns of one population? Program Objectives Program Objectives What objectives have been set for your program? What objectives have been set for your program? Stage of Development Stage of Development Is this a new initiative or is it well established? Is this a new initiative or is it well established?

13 12 Background and Description Resources Resources What resources are available to conduct the program activities? What resources are available to conduct the program activities? Activities Activities What are program staff doing to accomplish program objectives? What are program staff doing to accomplish program objectives? Outputs Outputs What are the direct and immediate results of program activities (materials produced, services delivered, etc.)? What are the direct and immediate results of program activities (materials produced, services delivered, etc.)? Outcomes Outcomes What are the intended effects of the program activities? What are the intended effects of the program activities? Reference: Table 3 in the Evaluation Plan Guide

14 13 Program Logic Model A logic model is a graphic depiction of the program description. Arrows describe the links between resources, activities, outputs and outcomes Arrows describe the links between resources, activities, outputs and outcomes A logic model A logic model Provides a sense of scope of your program Provides a sense of scope of your program Ensures that systematic decisions are made about what is to be measured Ensures that systematic decisions are made about what is to be measured Helps to identify and organize indicators Helps to identify and organize indicators

15 14 Program Logic Model

16 15 Contact Investigation Goal: Prevent TB among contacts to cases (by finding and testing contacts for TB and LTBI, and then treating infected contacts to completion). A Inputs B Activities C Short-term Outcomes D Intermediate Outcomes E Long-term Outcomes 1 Adequate infrastructure i Qualified, trained and and motivated staff ii Community and congregate setting partnerships iii Policies, procedures, and guidelines iv Ongoing data collection, monitoring, and reporting systems v Adequate physical, diagnostic, and treatment resources vi Linkages between jurisdictions viiAdequate data collection tools viii Partnership with private providers 1 a a Interview/reinterview cases i Build rapport ii Provide education iii Obtain information about source case and contacts b Locate and evaluate contact: i Follow-up ii Education iii Examination & testing* c Offer treatment d Treat contact – case management (DOT/DOPT/incentives 2 a Comprehensive interview tool b Staff trained in interview techniques 3 a Legal mandate to collect contact information from congregate settings 2 a Reporting 3 a Monitor: i Data collection ii Data management ii Data analysis iv Data dissemination a 4 Conduct periodic review of cases/contacts and progress toward contact treatment goals a Cases identify contacts 1 2 a Contacts educated 3 a Contacts evaluated 4 a Contacts followed up 5 a Contacts start treatment 6 a Evidence-based decisions about continuation or termination of contact investigation 2 a 1 a Contacts complete appropriate treatment for active TB or LTBI 1 a Active TB cured in contacts b TB (prevented) in contacts with LTBI 2 a Reduced incidence and prevalence of TB 3 a TB eliminated Improved approaches for contact investigation

17 16 Sample Logic Model Project Description of TB Support Program ResourcesActivitiesOutputsOutcomes InitialSubsequentShort-termIntermediateLong-term LHAs of Salvadoran community TB Staff Funding Community based organizations serving Salvadoran community Hiring LHAsLHAs trainingLHAs hired & trained Provide Spanish speaking and culturally competent services for Salvadoran community Increase utilization of TB services by Salvadoran Community Eliminate TB in Salvadoran community Community outreach EducationTB outreach and education conducted Increase TB knowledge in Salvadoran community Counseling & support provided Trust built between health care providers and the Salvadoran community TB screening/ testing Testing done and referrals made Early TB and LTBI detection and interventions Reduce TB transmission Patients offered treatment Develop treatment plans DOT visits Treatment plans DOT administered Patients accept treatment for TB and LTBI; Patients adhere to treatment Complete treatment Increase completion of therapy rate ( reduce hospital admissions for TB among Salvadorans)

18 17 Focus of the Evaluation Since you cannot feasibly evaluate everything, you must focus the evaluation by prioritizing and selecting evaluation questions. Stakeholder Needs Stakeholder Needs Who will use the evaluation findings? Who will use the evaluation findings? How will the findings be used? How will the findings be used? What do stakeholders need to learn/know from the evaluation? What do stakeholders need to learn/know from the evaluation?

19 18 Focus of the Evaluation Process Evaluation Process Evaluation What resources were required? What resources were required? What program activities were accomplished? What program activities were accomplished? Were they implemented as planned? Were they implemented as planned? Outcome Evaluation Outcome Evaluation Is the program producing the intended outcomes? Is the program producing the intended outcomes? Is there progress toward program objectives and goals? Is there progress toward program objectives and goals?

20 19 Focus of the Evaluation Evaluation Questions Evaluation Questions Based on the needs of your stakeholders Based on the needs of your stakeholders Address process and outcome Address process and outcome Assess Your Questions Assess Your Questions Feasible to collect Feasible to collect Provide accurate results Provide accurate results

21 20 Focus of the Evaluation Key Issues in Evaluation Design Key Issues in Evaluation Design Will you have a comparison or control group? Will you have a comparison or control group? When will you collect data? When will you collect data? Will the data be collected retrospectively or prospectively? Will the data be collected retrospectively or prospectively? What type of data do you need? What type of data do you need? What data do you have already? What data do you have already?

22 21 Focus of the Evaluation Other Design Considerations Other Design Considerations Standards for good evaluation Standards for good evaluation Timeliness Timeliness Stage of development Stage of development Data needed Data needed Strengthen Your Design Strengthen Your Design Mix methods whenever possible Mix methods whenever possible Use repeated measures Use repeated measures Triangulate Triangulate

23 22 Gathering Credible Evidence: Data Collection Identify indicators, standards, and data sources to address evaluation questions. Indicators Indicators Visible, measurable signs of program performance Visible, measurable signs of program performance Reflect program objectives, logic model and evaluation questions Reflect program objectives, logic model and evaluation questions Program Benchmarks and Targets Program Benchmarks and Targets Reasonable expectations of program performance Reasonable expectations of program performance Benchmarks against which to measure performance Benchmarks against which to measure performance Reference: Table 4 in your Evaluation Plan Guide

24 23 EvaluationQuestion Process and Outcome Indicators Program Benchmarks Have Spanish-speaking persons been treated appropriately for LTBI or TB? Number of Spanish-speaking persons treated by clinic for TB & LTBI between Jan – June. Increase in the number of Spanish- speaking patients Number of times clinical treatment standards are met for Spanish-speaking patients Clinical standards are met 100% of time Percent of time that signs and forms are available in Spanish and written for persons with low-literacy skills Patient education signs and forms in Spanish are available 100% of time; literacy level of materials is at a 3 rd grade reading level Linking evaluation questions, indicators and program benchmarks. Example from the Guide – Table 4. Gathering Credible Evidence: Data Collection

25 24 Gathering Credible Evidence: Data Collection Data Collection Data Collection Where are the data? Where are the data? What methods will be used to collect data? What methods will be used to collect data? How often will the data be collected? How often will the data be collected? Who will collect the data? Who will collect the data? Tools for Data Collection Tools for Data Collection Collect only the information you need Collect only the information you need Easy to administer and use Easy to administer and use Reference: Table 5 in your Evaluation Plan Guide

26 25 Table 5. Data Collection Plan Indicator Data Sources Collection WhoWhenHow Gathering Credible Evidence: Data Collection Linking indicators and data sources and specifying your data collection plan. Example from the Guide – Table 5.

27 26 Gathering Credible Evidence: Data Collection Human Subjects Considerations Human Subjects Considerations Evaluation Timeline Evaluation Timeline Ensures that all stakeholders are aware of what activities are occurring at any time Ensures that all stakeholders are aware of what activities are occurring at any time Helps to determine if your evaluation resources will be strained by too many activities happening at once Helps to determine if your evaluation resources will be strained by too many activities happening at once Data Management and Storage Data Management and Storage Ensures confidentiality and data quality Ensures confidentiality and data quality Reference: Table 6 in your Evaluation Plan Guide

28 27 Justifying Conclusions: Analysis and Interpretation Once the data are collected, analysis and interpretation will help you understand what the findings mean for your program. Analysis Analysis What analysis techniques will you use for each data collection method? What analysis techniques will you use for each data collection method? Who is responsible for analysis? Who is responsible for analysis? Interpretation Interpretation What conclusions will you draw from your findings? What conclusions will you draw from your findings? How will you involve stakeholders? How will you involve stakeholders? Reference: Table 7 in your Evaluation Plan Guide

29 28 Ensuring Use and Sharing Lessons Learned: Reporting and Dissemination A plan for dissemination and use of the evaluation findings will avoid having evaluation reports sit on the shelf. Dissemination Dissemination What medium will you use to disseminate findings? What medium will you use to disseminate findings? Who is responsible for dissemination? Who is responsible for dissemination? Use Use How, where, and when will findings be used? How, where, and when will findings be used? Who will act on the findings? Who will act on the findings? Reference: Table 8 in your Evaluation Plan Guide

30 29 Tips for Evaluation Planning Start small – focus on one initiative or program component to start with and limit the number of evaluation questions Start small – focus on one initiative or program component to start with and limit the number of evaluation questions Use what you already know about the program Use what you already know about the program Consider existing sources of data Consider existing sources of data Be realistic in your timeline and assessment of resources Be realistic in your timeline and assessment of resources Use the template and tables provided in the guide, adapt as needed Use the template and tables provided in the guide, adapt as needed Seek help with your evaluation Seek help with your evaluation

31 30 Evaluation Resources Some Web-Based Resources Centers for Disease Control and Prevention: Centers for Disease Control and Prevention: W.K. Kellogg Foundation: W.K. Kellogg Foundation: University of Wisconsin Extension: University of Wisconsin Extension: Selected Publications Connell JP, Kubisch AC, Schorr LB, Weiss, CH. New Approaches to Evaluating Community Initiatives, New York, NY: Aspen Institute, Connell JP, Kubisch AC, Schorr LB, Weiss, CH. New Approaches to Evaluating Community Initiatives, New York, NY: Aspen Institute, Patton MQ, Utilization-focused Evaluation, Thousand Oaks, CA: Sage Publications, Patton MQ, Utilization-focused Evaluation, Thousand Oaks, CA: Sage Publications, Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Approach. Newbury Park, CA: Sage Publications, Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Approach. Newbury Park, CA: Sage Publications, Taylor-Powell E, Steele S, Douglas M. Planning a Program Evaluation. Madison, Wl: University of Wisconsin Cooperative Extension, Taylor-Powell E, Steele S, Douglas M. Planning a Program Evaluation. Madison, Wl: University of Wisconsin Cooperative Extension, 1996.


Download ppt "0 Developing An Evaluation Plan For TB Control Programs Developing An Evaluation Plan For TB Control Programs Division of Tuberculosis Elimination National."

Similar presentations


Ads by Google