Presentation is loading. Please wait.

Presentation is loading. Please wait.

Program Evaluation Webinar Series Part 2: “Getting Started and Engaging Your Stakeholders” Presented by: Leslie Fierro and Carlyn Orians.

Similar presentations


Presentation on theme: "Program Evaluation Webinar Series Part 2: “Getting Started and Engaging Your Stakeholders” Presented by: Leslie Fierro and Carlyn Orians."— Presentation transcript:

1 Program Evaluation Webinar Series Part 2: “Getting Started and Engaging Your Stakeholders” Presented by: Leslie Fierro and Carlyn Orians

2 Getting Started and Engaging Your Stakeholders Leslie A. Fierro, MPH TKCIS Contractor NCEH/CDC Carlyn Orians, MA, Battelle Centers for Public Health Research and Evaluation Presented November 20, 2008

3 Agenda Definition of evaluation. Difference between evaluation & research. CDC’s Evaluation Framework. The who, why, when & how of stakeholders. Tangible examples of engaging stakeholders.

4 What is the Purpose of Evaluation? The purpose of evaluation can be: social betterment program planning and decision making empowerment of individuals

5 Evaluation: Definition 1 Evaluation is the systematic investigation of the merit, worth, or significance of an object. -- Michael Scriven

6 Evaluation: Definition 2 Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy. -- Carol Weiss

7 Evaluation: Definition 3 Evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. -- Michael Patton

8 The Common Element Note that all 3 of these statements share a common element... …they all define evaluation as a systematic and formalized endeavor.

9 Summative Evaluations Summative evaluations seek to judge a program by asking, “Should this project be…” cancelled? continued? expanded?

10 Formative Evaluations Formative evaluations seek to use the evaluation findings. They ask the question: “Is the program being conducted as planned?” “Is the program doing well?” Formative evaluations seek to improve programs or policies.

11 The CDC Definition Evaluation is the systematic collection of information about the activities, characteristics and outcomes of the program to make judgments about the program, improve program effectiveness and/or inform decisions about future program development.

12 Research vs. Evaluation Specific to Research Specific to Evaluation Research and evaluation share methodologies but ask different questions. Commonalities

13 The Research Context Purpose To add to an existing body of theoretical knowledge. Typical Questions Does the theory hold in this situation? How can we improve the theory to make better predictions in every situation?

14 Research Findings test and improve theories. develop generalizable theories. theories apply across different settings, people, and times. Research is conducted to:

15 The Evaluation Context Purpose Judge the merit or worth of a program. Provide information for decision making. Typical Questions Is the program doing what was intended? What steps might be taken to improve it?

16 Evaluation Findings Evaluation results: are not usually generalizable. focus on a specific situation. evaluation of a single program in a particular context.

17 Research vs. Evaluation Evaluation asks: “Is this program working?” Research asks: “Will this program work across multiple settings?”

18 The Role of Researchers Researchers: Play a single role as content experts.

19 The Role of Researchers Researchers: Play a single role as content experts. Identify gaps in current knowledge.

20 The Role of Researchers Researchers: Play a single role as content experts. Identify gaps in current knowledge. Derive their own questions.

21 The Role of Researchers Researchers: Play a single role as content experts. Identify gaps in current knowledge. Derive their own questions. Perform their own research.

22 The Role of Evaluators Evaluators: Play multiple roles—facilitator, educator, scientific expert, etc.

23 The Role of Evaluators Evaluators: Play multiple roles—facilitator, educator, scientific expert, etc. Involve stakeholders.

24 The Role of Evaluators Evaluators: Play multiple roles—facilitator, educator, scientific expert, etc. Involve stakeholders. Collaborate to identify and prioritize questions.

25 “Research seeks to prove, evaluation seeks to improve.” M.Q. Patton In a Nutshell …

26 The Findings Must be Useful To improve a program, the findings must be useful! How? The stakeholders must be involved. The questions must be relevant to the program. The findings must be credible to key stakeholders.

27 Getting Started and Engaging Your Stakeholders The Who, When, Why, and How of Stakeholder Involvement in CDC’s Evaluation Framework

28 “Who” are the Stakeholders? Stakeholders are: people and/or organizations that are interested in the program, are interested in the results of the evaluation and/or have a stake in what will be done with the results of the evaluation.

29 Examples of Stakeholders Persons involved in program operations Example: asthma program staff and partners Persons served or affected by the program Example: participants in asthma interventions, asthma patients, families, caregivers, etc. Intended users of evaluation findings Example: policy makers, managers, administrators, advocates, funders, and others

30 Each Program is Different Develop a list of stakeholders at the start of any evaluation activity.

31 Which Stakeholders Matter Most? Review your list of stakeholders and think strategically about these questions: “Who do we need to…”

32 Which Stakeholders Matter Most? Review your list of stakeholders and think strategically about these questions: “Who do we need to…” enhance credibility?

33 Which Stakeholders Matter Most? Review your list of stakeholders and think strategically about these questions: “Who do we need to…” enhance credibility? implement program changes?

34 Which Stakeholders Matter Most? Review your list of stakeholders and think strategically about these questions: “Who do we need to…” enhance credibility? implement program changes? advocate for changes?

35 Which Stakeholders Matter Most? Review your list of stakeholders and think strategically about these questions: “Who do we need to…” enhance credibility? implement program changes? advocate for changes? fund, authorize, or expand the program?

36 “Why” Engage Stakeholders? Engaging stakeholders is an important part of the CDC Evaluation Framework.  Involving stakeholders may be a requirement of your program.

37 “Why” Engage Stakeholders? Engaging stakeholders is an important part of the CDC Evaluation Framework Stakeholders will add credibility.  If you want the results to be acted upon, they must be credible.

38 “Why” Engage Stakeholders? Engaging stakeholders is an important part of the CDC Evaluation Framework. Stakeholders will add credibility. Stakeholders may have resources to help.  Stakeholders may be able to contribute data, skills, analytical skills, etc.

39 “Why” Engage Stakeholders? Engaging stakeholders is an important part of the CDC Evaluation Framework. Stakeholders will add credibility. Stakeholders may have resources to help. Stakeholders may be critical to implementing or advocating for action based on the results.

40 “Why” Engage Stakeholders? Engaging stakeholders is an important part of the CDC Evaluation Framework. Stakeholders will add credibility. Stakeholders may have resources to help. Stakeholders may be critical to implementing or advocating for action based on the results. You will build trust and understanding among program constituents.  Involving stakeholders helps to reduce fear of the evaluation process.

41 “Why” Would Stakeholders Want to be Involved? Get answers to their questions.

42 “Why” Would Stakeholders Want to be Involved? Get answers to their questions. Learn about evaluation.

43 “Why” Would Stakeholders Want to be Involved? Get answers to their questions. Learn about evaluation. Influence the design and methods.

44 “Why” Would Stakeholders Want to be Involved? Get answers to their questions. Learn about evaluation. Influence the design and methods. Protect their constituents.

45 “Why” Would Stakeholders Want to be Involved? Get answers to their questions. Learn about evaluation. Influence the design and methods. Protect their constituents. Motivated to help program succeed.

46 CDC’s Evaluation Framework Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEPS StandardsUtilityFeasibilityProprietyAccuracy

47 “When” and “How” Can You Engage Stakeholders? You can use the CDC Evaluation Framework to engage stakeholders in: identifying and prioritizing evaluation questions, selecting credible sources, and developing a plan for action based on evaluation results.

48 “When” and “How” Can You Engage Stakeholders? You can engage stakeholders in every step of the evaluation process. But… you don’t have to engage stakeholders in all these ways in every evaluation.

49 “How” Depends on… Evaluator preference.

50 “How” Depends on… Evaluator preference. Stakeholder preference.

51 “How” Depends on… Evaluator preference. Stakeholder preference. Resources.

52 “How” Depends on… Evaluator preference. Stakeholder preference. Resources. Degree of trust or threats to credibility.

53 “How” Depends on… Evaluator preference. Stakeholder preference. Resources. Degree of trust or threats to credibility. If there is a high degree of mistrust, engage stakeholders early in the evaluation process. This helps ensure that the results are viewed as credible and are acted upon.

54 CDC’s Evaluation Framework Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEPS StandardsUtilityFeasibilityProprietyAccuracy

55 Who should be involved? Develop list of potential stakeholders. Decide which stakeholders are the most important to include. CDC’s Framework Step 1 Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEP 1 StandardsUtilityFeasibilityProprietyAccuracy Engage stakeholders

56 CDC’s Framework Step 2 Do stakeholders share a vision of what the program does and its intended outcomes? Diverse views? Similar views? Engage them early in the process. Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEP 1 StandardsUtilityFeasibilityProprietyAccuracy Describe the program

57 CDC’s Framework Step 3 What are the most pressing and important evaluation questions for stakeholders? What questions do they need answered to be able to take action? What methods are available and preferred? What will be considered “credible evidence”? Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEP 1 StandardsUtilityFeasibilityProprietyAccuracy Focus the evaluation design

58 CDC’s Framework Step 4 Gather evidence stakeholders will find credible. Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEP 1 StandardsUtilityFeasibilityProprietyAccuracy Gather credible evidence

59 CDC’s Framework Step 5 How do diverse stakeholders interpret the findings? May engage stakeholders in the analysis. Perhaps solicit their interpretation before results are finalized. Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEP 1 StandardsUtilityFeasibilityProprietyAccuracy Justify conclusions

60 CDC’s Framework Step 6 Which stakeholders can play a role in disseminating results or acting on findings? This is a critical step for stakeholder involvement. Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEP 1 StandardsUtilityFeasibilityProprietyAccuracy Ensure use and share lessons learned

61 Getting Started and Engaging Your Stakeholders Tangible Examples of Stakeholder Involvement in the Evaluation Process

62 Example #1 – Asthma and Home Environment in Low-Income Apts Intervention to improve indoor environment for children with asthma via following activities:

63 Example #1 – Asthma and Home Environment in Low-Income Apts Intervention to improve indoor environment for children with asthma via following activities: Provide education and training for apartment owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes.

64 Example #1 – Asthma and Home Environment in Low-Income Apts Intervention to improve indoor environment for children with asthma via following activities: Provide education and training for apartment owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes. Work with city officials to enhance existing housing code.

65 Example #1 – Asthma and Home Environment in Low-Income Apts Intervention to improve indoor environment for children with asthma via following activities: Provide education and training for apartment owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes. Work with city officials to enhance existing housing code. Promote smoke-free housing.

66 CDC’s Evaluation Framework Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEPS StandardsUtilityFeasibilityProprietyAccuracy

67 Step 1: Engage the Stakeholders Low-Income Housing Scenario Who are the stakeholders?

68 Step 1: Engage the Stakeholders Low-Income Housing Scenario Who are the stakeholders? affected families

69 Step 1: Engage the Stakeholders Low-Income Housing Scenario Who are the stakeholders? affected families those who receive training (e.g., owners, inspectors, maintenance personnel, tenants)

70 Step 1: Engage the Stakeholders Low-Income Housing Scenario Who are the stakeholders? affected families those who receive training (e.g., owners, inspectors, maintenance personnel, tenants) trainers

71 Step 1: Engage the Stakeholders Low-Income Housing Scenario Who are the stakeholders? affected families those who receive training (e.g., owners, inspectors, maintenance personnel, tenants) trainers city officials responsible for the housing code

72 Step 1: Engage the Stakeholders Low-Income Housing Scenario Who are the stakeholders? affected families those who receive training (e.g., owners, inspectors, maintenance personnel, tenants) trainers city officials responsible for the housing code smoke-free advocates

73 Step 1: Engage the Stakeholders Low-Income Housing Scenario Who are the stakeholders? affected families those who receive training (e.g., owners, inspectors, maintenance personnel, tenants) trainers city officials responsible for the housing code smoke-free advocates Stakeholders may include advocates as well as those resistant to change. It’s important to include a mix of views so you don’t get surprised at the end.

74 Step 2: Describe the Program Low-Income Housing Scenario Obtain stakeholders’ views on:

75 Step 2: Describe the Program Low-Income Housing Scenario Obtain stakeholders’ views on: What is the current indoor environment?

76 Step 2: Describe the Program Low-Income Housing Scenario Obtain stakeholders’ views on: What is the current indoor environment? What changes are both desired and seen as realistic?

77 Step 2: Describe the Program Low-Income Housing Scenario Obtain stakeholders’ views on: What is the current indoor environment? What changes are both desired and seen as realistic? What outcomes do they think are most important?

78 Step 2: Describe the Program Low-Income Housing Scenario Obtain stakeholders’ views on: What is the current indoor environment? What changes are both desired and seen as realistic? What outcomes do they think are most important? Which activities contribute to which outcomes?

79 Step 2: Describe the Program Low-Income Housing Scenario Obtain stakeholders’ views on: What is the current indoor environment? What changes are both desired and seen as realistic? Which outcomes do they think are most important? Which activities contribute to which outcomes? (You could use a logic model here… Stay tuned for the next Webinar in this series…)

80 Step 3: Focus the Evaluation Design Low-Income Housing Scenario What questions do the stakeholders think are the most important to ask?

81 Step 3: Focus the Evaluation Design Low-Income Housing Scenario What questions do the stakeholders think are the most important to ask? Should the evaluation focus on changes to the housing code?

82 Step 3: Focus the Evaluation Design Low-Income Housing Scenario What questions do the stakeholders think are the most important to ask? Should the evaluation focus on changes to the housing code? Focus on trigger reduction? Which triggers?

83 Step 3: Focus the Evaluation Design Low-Income Housing Scenario What questions do the stakeholders think are the most important to ask? Should the evaluation focus on changes to the housing code? Focus on trigger reduction? Which triggers? What is the best design?

84 Step 3: Focus the Evaluation Design Low-Income Housing Scenario What questions do the stakeholders think are the most important to ask? Should the evaluation focus on changes to the housing code? Focus on trigger reduction? Which triggers? What is the best design? Is a pre-post design best?

85 Step 3: Focus the Evaluation Design Low-Income Housing Scenario What questions do the stakeholders think are the most important to ask? Should the evaluation focus on changes to the housing code? Focus on trigger reduction? Which triggers? What is the best design? Is a pre-post design best? Should there be a control group?

86 Step 3: Focus the Evaluation Design Low-Income Housing Scenario What questions do the stakeholders think are the most important to ask? Should the evaluation focus on changes to the housing code? Focus on trigger reduction? Which triggers? What is the best design? Is a pre-post design best? Should there be a control group? Are observational methods appropriate?

87 Step 4: Gather Credible Evidence Low-Income Housing Scenario What role can stakeholders play?

88 Step 4: Gather Credible Evidence Low-Income Housing Scenario What role can stakeholders play? Are there stakeholders that can facilitate access to residences and/or translate if necessary?

89 Step 4: Gather Credible Evidence Low-Income Housing Scenario What role can stakeholders play? Are there stakeholders that can facilitate access to residences and/or translate if necessary? Can the trainers help collect data before, during, or after each training? Other opportunities for data collection?

90 Step 4: Gather Credible Evidence Low-Income Housing Scenario What role can stakeholders play? Are there stakeholders that can facilitate access to residences and/or translate if necessary? Can the trainers help collect data before, during, or after each training? Other opportunities for data collection? What information will the stakeholders need to act? For example, will policy makers be more influenced by information on environmental conditions, asthma symptoms, or medical care usage?

91 Step 4: Gather Credible Evidence Low-Income Housing Scenario What role can stakeholders play? Are there stakeholders that can facilitate access to residences and/or translate if necessary? Can the trainers help collect data before, during, or after each training? Other opportunities for data collection? What information will the stakeholders need to act? For example, will policy makers be more influenced by information on environmental conditions, asthma symptoms, or medical care usage? Data in what form? Pictures? Stories? Numerical data?

92 Step 5: Justify Conclusions Low-Income Housing Scenario How do stakeholders view success? What does “success” look like to them?

93 Step 5: Justify Conclusions Low-Income Housing Scenario How do stakeholders view success? What does “success” look like to them? Do stakeholders have alternative interpretations of the initial findings?

94 Step 5: Justify Conclusions Low-Income Housing Scenario How do stakeholders view success? What does “success” look like to them? Do stakeholders have alternative interpretations of the initial findings? Do they perceive barriers you might have missed?

95 Step 6: Ensure Use and Share Lessons Learned Low-Income Housing Scenario Can stakeholders help disseminate evaluation findings?

96 Step 6: Ensure Use and Share Lessons Learned Low-Income Housing Scenario Can stakeholders help disseminate evaluation findings? Is there an owner’s newsletter? A tenant meeting? A neighborhood association?

97 Step 6: Ensure Use and Share Lessons Learned Low-Income Housing Scenario Can stakeholders help disseminate evaluation findings? Is there an owner’s newsletter? A tenant meeting? A neighborhood association? Is information needed in a different form to prompt action? A report? An executive brief? An oral presentation?

98 Step 6: Ensure Use and Share Lessons Learned Low-Income Housing Scenario Can stakeholders help disseminate evaluation findings? Is there an owner’s newsletter? A tenant meeting? A neighborhood association? Is information needed in a different form to prompt action? A report? An executive brief? An oral presentation? Is there interest beyond the community involved in the intervention?

99 What if you ignored stakeholders? If you ignore the stakeholders, potential pitfalls include…

100 What if you ignored stakeholders? Failure to ask important questions. For example, it may be important to know if tenants have access to quit-smoking services. If you ignore the stakeholders, potential pitfalls include…

101 What if you ignored stakeholders? Difficulty gaining access to or trust of program constituents. For example, will you gain access to data or be received into residents’ homes? If you ignore the stakeholders, potential pitfalls include…

102 What if you ignored stakeholders? Failure to uncover unanticipated consequences. For example, if improvements are made, will rents increase? If you ignore the stakeholders, potential pitfalls include…

103 What if you ignored stakeholders? Failure to communicate results in ways that are understood and/or accessible. For example, are there literacy or language issues? Cultural preferences? If you ignore the stakeholders, potential pitfalls include…

104 What if you ignored stakeholders? If you fail to involve the stakeholders… …your evaluation may not lead to action!

105 105 Example #2 – Care Coordination Across Health Systems Intervention to provide and integrate care coordination and case management for high-risk children with asthma.

106 106 Example #2 – Care Coordination Across Health Systems Intervention to provide and integrate care coordination and case management for high-risk children with asthma. Involves standardizing protocols across care systems, including: Medicaid HMOs Home nursing agencies Health departments

107 107 Example #2 – Care Coordination Across Health Systems Intervention to provide and integrate care coordination and case management for high-risk children with asthma. Involves standardizing protocols across care systems, including: Medicaid HMOs Home nursing agencies Health departments The goal: to prove success and thus convince insurers to continue reimbursing case management services.

108 Engage the Stakeholders Housing ScenarioHealthcare Scenario affected families those who receive training trainers city officials smoke-free advocates Who are the stakeholders in this scenario?

109 Engage the Stakeholders Housing ScenarioHealthcare Scenario affected families those who receive training trainers city officials smoke-free advocates insurers

110 Engage the Stakeholders Housing ScenarioHealthcare Scenario affected families those who receive training trainers city officials smoke-free advocates insurers case managers

111 Engage the Stakeholders Housing ScenarioHealthcare Scenario affected families those who receive training trainers city officials smoke-free advocates insurers case managers families

112 Engage the Stakeholders Housing ScenarioHealthcare Scenario affected families those who receive training trainers city officials smoke-free advocates insurers case managers families health system administrators

113 Focus the Evaluation Design Housing ScenarioHealthcare Scenario What changes are both desired and seen as realistic? What outcomes do they think are most important? Which activities contribute to which outcomes? What outcome questions?

114 Focus the Evaluation Design Housing ScenarioHealthcare Scenario What changes are both desired and seen as realistic? What outcomes do they think are most important? Which activities contribute to which outcomes? Is this intervention sustainable?

115 Focus the Evaluation Design Housing ScenarioHealthcare Scenario What changes are both desired and seen as realistic? What outcomes do they think are most important? Which activities contribute to which outcomes? Is this intervention sustainable? What is the cost of doing this kind of case management?

116 Gather Credible Evidence Housing ScenarioHealthcare Scenario Data in what form? Pictures? Stories? Numerical data? Stakeholders that can facilitate access ? Trainers help collect data ? What type of data do they need to make a decision?

117 Gather Credible Evidence Housing ScenarioHealthcare Scenario Data in what form? Pictures? Stories? Numerical data? Stakeholders that can facilitate access ? Trainers help collect data ? Is quantitative data more highly valued than qualitative data?

118 Gather Credible Evidence Housing ScenarioHealthcare Scenario Data in what form? Pictures? Stories? Numerical data? Stakeholders that can facilitate access ? Trainers help collect data ? Is quantitative data more highly valued than qualitative data? Do the data need to show cost savings?

119 Gather Credible Evidence Housing ScenarioHealthcare Scenario Data in what form? Pictures? Stories? Numerical data? Stakeholders that can facilitate access ? Trainers help collect data ? Be sensitive to the stakeholders’ interests and time constraints.

120 Gather Credible Evidence Housing ScenarioHealthcare Scenario Data in what form? Pictures? Stories? Numerical data? Stakeholders that can facilitate access ? Trainers help collect data ? Be sensitive to the stakeholders’ interests and time constraints. Be aware of and consider the political environment.

121 Example #3 – Daycare Education Intervention to train childcare providers to identify triggers and to manage children with asthma.

122 Example #3 – Daycare Education Intervention to train childcare providers to identify triggers and to manage children with asthma. Intervention reaches out to: Large daycare centers Licensed home daycare programs Participation is encouraged through continuing education credits

123 Example #3 – Daycare Education Intervention to train childcare providers to identify triggers and to manage children with asthma. Intervention reaches out to: Large daycare centers Licensed home daycare programs Participation is encouraged through continuing education credits Long-term goal: to expand beyond pilot providers to larger community.

124 Engage the Stakeholders Housing ScenarioDaycare Scenario affected families those who receive training trainers city officials smoke-free advocates Who would be the stakeholders in this scenario?

125 Engage the Stakeholders Housing ScenarioDaycare Scenario affected families those who receive training trainers city officials smoke-free advocates trainers daycare providers licensing boards families

126 Focus the Evaluation Design Housing ScenarioDaycare Scenario What changes are both desired and seen as realistic? What outcomes do they think are most important? Which activities contribute to which outcomes? What outcome questions?

127 Focus the Evaluation Design Housing ScenarioDaycare Scenario What changes are both desired and seen as realistic? What outcomes do they think are most important? Which activities contribute to which outcomes? behavior change environmental outcomes evidence to support decision to expand beyond pilot

128 Focus the Evaluation Design Housing ScenarioHealthcare Scenario Focus on changes to the housing code? Trigger reduction? Should there be a control group? Observational methods? What design mix?

129 Focus the Evaluation Design Housing ScenarioHealthcare Scenario Focus on changes to the housing code? Trigger reduction? Should there be a control group? Observational methods? Consider pre-post design. Consider control group design.

130 Gather Credible Evidence Housing ScenarioHealthcare Scenario Data in what form? Pictures? Stories? Numerical data? Stakeholders that can facilitate access ? Trainers help collect data ? What data collection methods?

131 Gather Credible Evidence Housing ScenarioHealthcare Scenario Data in what form? Pictures? Stories? Numerical data? Stakeholders that can facilitate access ? Trainers help collect data ? Success stories may be important.

132 Getting Started and Engaging Your Stakeholders Tangible Examples of Stakeholder Involvement in Applying the Evaluation Standards

133 Evaluation Standards Apply to Every Step Ensure use and share lessons learned Gather credible evidence Engage stakeholders Describe the program Focus the evaluation design Justify conclusions STEPS StandardsUtilityFeasibilityProprietyAccuracy

134 The Evaluation Standards Standards UtilityUtility FeasibilityFeasibility ProprietyPropriety AccuracyAccuracy

135 The Evaluation Standards There are actually 30 evaluation standards grouped into four categories. A complete list of the standards is published in CDC’s Framework for Program Evaluation in Public Health. (See link under “Learning Aids”.)

136 Why Use Standards? Standards provide a way to: Make difficult decisions when designing and implementing an evaluation. J udge the quality of an evaluation. D etermine where an evaluation can be better balanced.

137 The “Utility” Standard Who needs the evaluation information and what information do they need? Identify affected stakeholders. Collect data that addresses important questions. Ensure the data is responsive to the needs of the stakeholders.

138 The “Feasibility” Standard How much money, time, and effort can we put into this? Keep the burden of data collection as low as possible. Consider the political interests of the stakeholders. Ensure that the evaluation is cost effective.

139 The “Propriety” Standard What steps need to be taken for the evaluation to be ethical? Ensure the ethical treatment of evaluation participants. Ensure the ethical treatment of those who commissioned the evaluation. Ensure the ethical treatment of those affected by the findings of the evaluation.

140 The “Accuracy” Standard What design will lead to accurate information? Clearly and accurately document the program. Gather data in a way that produces reliable information. Analyze and report the data in a systematic and unbiased way. Assure that the source of the information is defensible.

141 CDC’s Asthma Control Program CDC’s Asthma Control Program funds 35 states and territories to implement statewide asthma control programs using a public health perspective. Their approach involves: Engaging, enhancing, and maintaining relationships with partners. Developing, improving, and conducting asthma surveillance activities. Designing and implementing interventions with partners.

142 Example #4 - Asthma Program Monitoring System This example involves collecting information for the purpose of evaluation across multiple sites.

143 Example #4 - Asthma Program Monitoring System Purpose: Develop a strategic, systematic approach to collecting information about our program activities, progress, and accomplishments across all funded states.

144 Employing the Framework Model Step 1: Involve the stakeholders CDC Air Pollution and Respiratory Health Branch. Representatives of funded State Asthma Programs. Step 2: Describe the Program Develop logic models. Step 3: Focus the evaluation design Identify and prioritize evaluation questions. Step 4: Gather credible evidence Create data collection instrument.

145 Developing the Data Collection Instrument In this example, developing the data collection instrument was a highly collaborative endeavor aided by the program evaluation standards.

146 The Iterative Process CDC internal workgroup identifies core information needs and drafts survey instrument. State workgroup members review instrument, pilot test, provide pilot data and comments. Create mock report to illustrate type of information obtained from instrument. CDC internal workgroup discusses comments and mock report. Modify instrument as necessary.

147 Applying the Standards How were the evaluation standards employed in this example? How was “engaging the stakeholders” important in applying each of the standards?

148 The “Utility” Standard Who are the intended users of the information? Why do they need the information? CDC Air Pollution and Respiratory Health Branch State partners Examine progress. Identify promising patterns. Share successful practices Identify areas for improvement. Compare and contrast program designs and performance.

149 The Utility Standard in Action Determined what information the CDC needed to do their job and how they will use the information. Engaged states in discussions about how to make the information useful for them. Developed mock reports to facilitate conversations with stakeholders about how the data could be used.

150 The “Feasibility” Standard How much money, time, and effort can we put into this? Remained conscious of how much time this data collection will require of the states. Identified areas where processes could be streamlined. Recognized that there will be less time for other programmatic activities.

151 The Feasibility Standard in Action Carefully selected “need-to-know” questions. Asked states: What information do you already have? How feasible is it for you to obtain? How long does it take to locate information? How long does it take to fill out the data collection instrument? Always kept the burden of data collection as low as possible.

152 The Feasibility Standard in Action The content of the data collection instrument was a careful balance between feasibility and utility. High priority data requiring higher burden was balanced by eliminating lower priority questions.

153 The “Propriety” Standard What steps need to be taken for the evaluation to be ethical? Program monitoring systems can be used to generate performance measures. The data collected could have significant implications. Understand that the information collected can affect the stakeholders’ programs.

154 The Propriety Standard in Action Used a highly collaborative process. Fostered open, honest dialogue about expectations and concerns. Remained highly responsive to stakeholder concerns. Modified instrument to address stakeholder concerns.

155 The “Accuracy” Standard What design will lead to accurate information? Be certain that the information collected is valid and reliable. If the information is inconsistent or irrelevant to the concept, reconsider whether to collect it.

156 Balancing the Standards When balancing feasibility vs. accuracy consider the effect on propriety.

157 The Accuracy Standard in Action Involved states in: Pilot tests and reviews. Discussions about “guessing”. Analyzing the quality of the information provided. Revising questions or collecting information that would help to improve accuracy. Decisions to discard some questions.

158 What We Have Reviewed Today The goal of evaluation and role of stakeholders. Who to engage as stakeholders. Why it is important to engage stakeholders. When and how to engage stakeholders. Stakeholder involvement across all steps of CDC framework. Stakeholder involvement in fulfilling the evaluation standards. Examples: Single and multiple sites

159 Thank You for Joining Us! Leslie Fierro: Carlyn Orians:


Download ppt "Program Evaluation Webinar Series Part 2: “Getting Started and Engaging Your Stakeholders” Presented by: Leslie Fierro and Carlyn Orians."

Similar presentations


Ads by Google