Presentation on theme: "Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good Identification and formulation of questions is a critical phase of the evaluation."— Presentation transcript:
1Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good Identification and formulation of questions is a critical phase of the evaluation processThe essence of evaluation is generating credible answers to questions about the performance if a social program.Questions must be answerable with the resources available
2Evaluation revolves around the hub of carefully crafted evaluation questions Questions need to be meaningful for stakeholders and program decision makersWho will use the the evaluation results?What info do they need?How do they expect to use it?The evaluator’s own analysis of the program is also important
3Communication with decision makers and stakeholders Questions are formulated through discourse and negotiationEngaging these groups increases their understanding and appreciation, and makes effective use of the findings when availableEvaluators can’t depend solely on their perspectives – they are not experts on evaluation
4Role of the EvaluatorFraming the questions – the evaluator knows how to analyze a program and focus an evaluation.Raise issues that might be overlookedIdentify aspects of program’s operation and outcomes that warrant inquiryDraw out concerns from stakeholders that can be translated into questions that can be answered by evaluation research
5Written summary of questions Helps evaluator guide the evaluation design. This is a useful reference for design and selecting research procedures.Can be shared with stakeholders to ensure their concerns are addressed.Safeguards against later misunderstanding of what the evaluation was suppose to accomplish.
6The 2 most important topics related to evaluation questions: 1. How to formulate evaluation question that can be addressed using research procedures available to the evaluator2. How to determine specific questions that the evaluation should focus.
7What makes a good evaluation question? Focus the evaluation on the areas of program performance at issue for key decision makers and stakeholdersFacilitate design of data collection to provide meaningful info about performanceIdentify a distinct dimension of program performance that can be credibly assessed
8What it means to evaluate something Establishing criteria of merit – on what dimension must the evaluand do well?Constructing standards – how well should should the evaluand perform?Measuring performance and comparing with standards – How well did the evaluand perform?Synthesizing and integrating data into a judgment of merit or worth. What is the merit or worth of the evaluand
9……a thoughtTo evaluate anything means to assess the merit or worth of something against criteria and standards.
10Dimensions of Program Performance Good evaluation questions must first be reasonable and appropriateThe questions must be answerable – specific, concrete, practical, and measurable
11Reasonable and Appropriate Questions Work with stakeholders to scale down and focus the questions –Lit Review – social science and social service literatureGet acquainted with the program – its structure, activities, roles and tasks of personnel, nature of participants, and assumptions.
12Questions must be Answerable Easy to formulate unanswerable questions without realizing it….Does this program enhance family values?Must be possible to identify some evidence or “observables” that can be obtained and will be credible as the basis for an answer.Have measurable performance dimensions stated in unambiguous terms with non-controversial definitions.
13Criteria for Program Performance What distinguishes evaluation questions is they have to do with performance and are associated with criteria by which performance can be judged.Performance Criterion (or standard) that apply, as well as the performance dimension at issue.An evaluation that only describes performance, but doesn’t assess it, is not truly evaluation.
14Standards by which program performance may be evaluated: Needs/wants of target populationProgram goals and objectivesProfessional standardsCustomary practice, normsLegal requirementsEthical/moral values: social justice, equity
15Standards…continued Past performance; historical data Targets set by program managersExpert opinionPre-intervention baseline levels for target populationConditions expected in the absence of the programCost or relative cost
16Places to look for performance criteria: Professional standards – esp in medical and health programsPrior experienceEvaluation and program literatureJudgement ratings from stakeholders to establish criterion levels or ranges (low to high performance)
17Typical evaluation Questions Need for program services (needs assessment)Program’s conceptulization or design (assessment of program theory)Program operations and service delivery (assessment of program process)Program outcomes (impact assessment)Program cost and efficiency (efficiency assessment)
18Order of the five categories or questions The questions draw meaning from the answers to the prior questions.Implementation failure – poor outcomes due to program activities needed to bring about desired improvements did not actually occur. Nutritional status of homeless people did not improve because soup kitchen was rarely open.Theory failure- Program conceptualization and design can’t generate desired outcomes no matter how well implemented – soup kitchen was far away.
19Evaluation Hierarchy – Assessment of: 1. Need for the Program2. Program Design and Theory3. Program Process and Implementation4. Program Outcome?Impact5. Assessment of Program Cost and Efficiency
20When developing questions Start at the bottom of the evaluation hierarchy and consider what is known and what needs to be known about the issuesThen move up the ladderLogical interdependencies between the levelsPremature attention to higher-order evaluation questions can be avoided
21Specific Questions Evaluators Should Ask Don’t rely only on input from sponsors or stakeholders – they are familiar with the program and can overlook critical, routine aspects or performance.Evaluator should make an independent analysis of program performance pertinent for investigation
22continued Multiple stakeholders = multiple views Set priorities with them and integrate as many relevant concerns as possibleIf they have constraints, the evaluator must decide whether or not to conduct the evaluation anyway or negotiate to broaden additional perspectives
23EssentialStrive to ensure stakeholders understand and accept the nature of the eval process, the type of info it will produce, what it might mean if results come out one way or another, and what ambiguities or unanswered questions remain.
24Snowball approachIdentify and contact a person or group, they then nominate others, who in turn nominate others.The process ends when no new nominations are given.
25Topics to discuss with Stakeholders Why the evaluation is neededThe program goals and objectivesWhat are the most important questions for the evaluation to answer?
26Backward mappingStart with a specification of the desired endpoint and work backward to determine what must be done to get there.
27Analysis of Program Assumptions and Theory Construct a conceptual model of how the program is expected to work and the connections presumed between activities and functions and social benefits it is intended to produce.Program theory is the set of assumptions about relationships between strategy and tactics and the social benefits it expects to produce.Program theory is also called program conceptulaization, program plan, blueprint or design.
28Logic ModelLays out expected sequence of steps going from program services to client outcomes.
29Collating Eval Questions and Setting Priorities Questions tend to cluster around program functions – ex services or outcomesAlso around issues – ex need, design, impactWhen the evaluator prioritizes the questions for a selected program, the next phase is design.
30Summary Assess the need for a program Program theory or plan for addressing the needImplementation of the program and associated processImpact or outcome of the program implementation on the social needEfficiency with which the program attains its outcomes