Overview of Chapters 11 – 13, & 17

Clarifying the evaluation request and responsibilities Chapter 11 Clarifying the evaluation request and responsibilities An organization has requested that you to complete an evaluation. What questions do you feel are important to ask before beginning the evaluation? Are there times when you would decline a request for evaluation? Why? What are some of the advantages and disadvantages in having an evaluation conduct by an external evaluator? By an internal evaluator? (take 5 minutes with a partner to discuss)

Clarifying the Evaluation Request and Responsibilities Identify the stakeholders Key questions: Who are the sponsors and clients? Who are the managers and staff? Are there other interest groups, such as agencies, elected officials, the public at large? 2. Identify the purpose for the evaluation Why is the evaluation being requested? What question will it answer? How will the results be used? By whom?

Clarifying the Evaluation Request and Responsibilities 3. Determine if the Evaluation is Appropriate Key questions: How will the results be used? Will the results produce trivial information? Is the program too new? Is the evaluation purpose ethical?

Clarifying the Evaluation Request and Responsibilities Steps to follow: Clarify the intended program model or theory Examine the program in implementation to determine whether it matches the model and could achieve the goals or objectives Explore different approaches to determine if they meet the stakeholders information needs Agree on evaluation priorities and intended uses of the study

Who will evaluate the program? Internal vs. External evaluator Internal: know the organization, history, and decision-making style and would be around to encourage use of evaluation results External: bring a greater objectivity and sometimes specialized skill for a particular project or evaluation

Chapter 12 Setting Boundaries and Analyzing the Evaluation Context 1. After determining who the stakeholders are, how do you determine who should be involved in the evaluation process? 2. What should the evaluator consider in analyzing the political context in which an evaluation will occur? (take 5 minutes to discuss with a partner)

Setting Boundaries and Analyzing the Evaluation Context The evaluator should identify and communicate with each stakeholder group or its representative: 1. To learn about the groups concerns 2. To understand how they will use the evaluation results This will help the evaluator weigh their input during the evaluation

Setting Boundaries and Analyzing the Evaluation Context Do evaluators need to be aware of Squeaking wheels &Powerful Stakeholders? You need to bring about a “democratic dialogue” if possible, with a balance in the number of people involved and diversity within the group.

Setting Boundaries and Analyzing the Evaluation Context Greene (2005) identified four groups: People who have decision authority over the program People who have direct responsibility for the program People who are the intended beneficiaries of the program

Setting Boundaries and Analyzing the Evaluation Context Scriven (2007) adds to the list: Political supporters or opponents Community Leaders Public in General

Setting Boundaries and Analyzing the Evaluation Context Political Context: 1. Who will gain to gain/lose? 2. Which individuals and groups have power in the setting? 3. How should the evaluator relate/communicate to the groups? 4. Will all groups cooperate? 5. Which groups have a vested interest in the outcomes? 6. What safeguards should be incorporated into the evaluation agreement?

Chapter 13 Identifying and selecting the evaluation questions and criteria What is the focus of evaluation questions? What are good sources for evaluation questions? Who should be involved in the divergent and convergent phases? Why?

Identifying and selecting the evaluation questions and criteria Evaluation Questions: Provide focus to the evaluation Specify the information the evaluation will provide Guide choices for data collection, analysis, and interpretation

Identifying and selecting the evaluation questions and criteria Good Sources: Existing Standards in the Field Research Literature Content Experts Evaluators Own Experience

Identifying and selecting the evaluation questions and criteria Divergent: Cronbach (1982), “Opening one’s mind to be entertained at least briefly as prospects for investigation” (p. 210). Fitzpatrick, et al. (2011), “throw a broad net and learn from many possible sources” (p. 316).

Identifying and selecting the evaluation questions and criteria Convergent: Three reasons for reducing the range of variables. Budget Manageability/Complexity Attention Span of Audience The evaluator aims for the maximum bandwith. Cronbach (1982) *Typically completed by an advisory group with representatives from different stakeholder groups.

Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding What considerations are important in tailoring the reporting of evaluation results to audiences? What are some of the ways results can be communicated to stakeholders? What should be included in an evaluation report? (take 5 minutes to discuss with a partner)

Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding Different audiences have different information needs and knowledge levels. Reading ability Familiarity with the program Attitude toward the program Role in decision making Familiarity with evaluation practices/methods Attitude and interest in evaluation Experience in using evaluation results Torres, Preskill, and Pointek (2005)

Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding If possible (Patton, 2008) points out that evaluation data are sued more if the evaluator discusses and negotiates the format, style, and organization of reports with primary users. Other Key Points: Avoid Jargon, Use simple direct language, Use examples, anecdotes, be interesting…

Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding Key Elements of an Evaluation Report (p. 470) Executive Summary Introduction to the Report Focus of the Evaluation Brief Overview of Evaluation Plan & Procedures Presentation of Results Conclusion and Recommendations Minority Reports or Rejoinders Appendices