Presentation on theme: "Designing an Effective Evaluation Strategy"— Presentation transcript:
1 Designing an Effective Evaluation Strategy 6/30/2005Designing an Effective Evaluation StrategyThis is the introduction to the program. The goal you are trying to achieve is to relax the participants and make them feel they are part of the process.It is good to use some type of ice breaker where everyone talks about themselves.Since time is limited, you can have people give their name, job, agency and why they are here.Then using easel pads – list what everyone says even if a duplication.Post the sheets on the wall where everyone can see them and go through the items. Note what you are going to do and what you aren’t going to cover. This will keep participant expectations.If you aren’t going to cover something and know where they can go to get the information tell them.GO TO NEXT SLIDE AND SPECIFICALLY LIST THE GOALS TO BE ACCOMPLISHEDpresenter: Sharon Schnelle, Ph.D.Sponsored throughSuccessful Grant Writing
2 Goals of Training Participants will understand 6/30/2005Goals of TrainingParticipants will understandThe terminology related to evaluationEvaluation design options and how to plan out a successful evaluation strategyHow to design an effective evaluation strategyData collection sources and methodsHow to justify conclusions based on evaluation resultsProgram Planning – We will be discussing how to plan for a grant. You should start to plan for an activity long before you consider applying for a grant. Create a notebook and put all the information you collect and write in the notebook. Then when you go to actually write the grant you can get information out as appropriate for the grant.Problem Statement & Objectives – These two items are probably the most critical in the application and usually the poorest written. We will be spending quite a lot of time on these two items so you can understand how to put them together. From a reviewers perspective, it you can’t get these right, then you don’t deserve the grant. You probably don’t know what you want to accomplish.Organizational Credibility – How your organization is perceived is critical your ability to actually obtain funding. We will be discussing what your organization can do to improve the perception of grantors.Prepare Proposals – As we talk today, I will be giving you suggestions on what needs to be in a grant application and how reviewers respond to grants. This will be practical advise on how to put the proposal together.Funding Sources – Here’s probably what you all came to hear [this is usually the case but change based on what expectations were] but we will be leaving this to the last thing. This is one way to keep you hear! Seriously, this is last because all of the other parts of the process need to be completed before you start looking for funding.Successful Grant Writing
3 What is “evaluation”?The simple answer, dictionary definition:e + valuer =to establish the worth or value of.Evaluation can be described as the systematic investigation of the merit, worth, or significance of any “object”
4 Program Evaluation defined 6/30/2005Program Evaluation definedEvaluation is the systematic application of scientific methods to assess the design, implementation, improvement or outcomes of a program (Rossi & Freeman, 1993; Short, Hennessy, & Campbell, 1996). The term "program" may include any organized action such as media campaigns, service provision, educational services, public policies, research projects, etc. (Center for Disease Control and Prevention [CDC], 1999).Letters of participation are used to determine support for the project and the authors involvement in the project. They are critical to the success of any review of the grant.What are reviewers looking forSuccessful Grant Writing
5 6/30/2005Why do we do Evaluation?Establish model programs and best practices by providing feedback about what worked and what failedTool of good management and quality improvement - gain insight into effective strategies on how to improve performanceMeasures impact the program is makingRequired by funderSuccessful Grant Writing
6 Purpose of Program Evaluation 6/30/2005Purpose of Program EvaluationDemonstrate program effectiveness to fundersImprove the implementation and effectiveness of programsBetter manage limited resourcesDocument program accomplishmentsJustify current program fundingSupport the need for increased levels of fundingSatisfy ethical responsibility to clients to demonstrate positive and negative effects of program participation (Short, Hennessy, & Campbell, 1996).Document program development and activities to help ensure successful replicationSuccessful Grant Writing
7 Effective evaluation is not an "event" that occurs at the end of a project, but is an ongoing process which helps decision makers better understand the project; how it is impacting participants, partner agencies and the community; and how it is being influenced/impacted by both internal and external factors.W.K. Kellogg Foundation Evaluation Handbook, p. 3
9 Framework for Evaluation 6/30/2005Framework for EvaluationSuccessful Grant Writing
10 Underlying Logic of Steps No evaluation is good unless… results are used to make a differenceNo results are used unless… a market has been created prior to creating the productNo market is created unless…. the evaluation is well-focused, including most relevant and useful questions
11 Program Planning Process 6/30/2005Program Planning ProcessEvaluation is part of a larger program planning process. First, you plan the program. Then you implement it. Then you evaluate it. You use what you learn from the evaluation to improve your program. And then you start planning your improved program.Successful Grant Writing
12 Evaluation Need not be Some evaluation is better than none 6/30/2005EvaluationNeed not beExpensiveComplicatedTime consumingSome evaluation is better than noneExternal evaluator is sometimes seen as more objective than internalEvaluator should be qualifiedEvaluation plan should be meaningful, related to goals and objectives, and be an honest examination of programSuccessful Grant Writing
13 Continuous Quality Improvement (CQI) cycle. These must be integrated…Planning—What actionswill best reach our goalsand objectives.Performance measurement—How are we doing?Evaluation—Why are wedoing well or poorly?
15 6/30/2005Types of EvaluationPlanning (involves examining the developmental issues prior to setup)Process or Formative (involves monitoring the “process,” ensuring activities are completed on time and on target, while the program is ongoing)Tells you if you’re on trackPoints to improvementOutcome or Summative (involves assessing the outcome at the conclusion of the program and measures how change that has occurred as a result of the program)Shows what impact you have on problemHelps justify programCost BenefitSuccessful Grant Writing
16 Planning Evaluation Why is the program needed? 6/30/2005Planning EvaluationWhy is the program needed?Identify target populationWho needs to be involved in the planning?Identify key players/agenciesWhat are the goals of the program?Identify goals from perspective of various key stakeholders/agenciesWhat resources are necessary?Identify financial resourcesIdentify non-financial resourcesWhat is the timeline?Determine timeline to program implementationSuccessful Grant Writing
17 Planning and Process Evaluation Goals 1)To examine developmental issues prior to setup2) To assess the steps that occur within the program set-up phase. Key questions to answered through this type of evaluation include:How was the program set-up initiated?What agencies are involved in its daily operation?How were collaborations developed, and how are they sustained?Overall, how does the program function to serve the clients?What resources (both financial and non-financial) are available/needed?What is the timeline for the project?
18 Process Evaluation Data – Types and Collection TYPES: (Which is more useful?)QuantitativeQualitativeHow to collect data for process evaluation:InterviewsFocus groupsObservationQuestionnaires/SurveysAnalysis of existing documents
19 Outcome EvaluationOutcome evaluations are sometimes referred to as “true” or “real” evaluationOutcome evaluation is no more “true” or “real” than process evaluationIndeed, a quality process evaluation is necessary if one is going to say what it is about the program that produced the results
20 Outcome Evaluation Data – Types and Collection The essence of program outcome evaluation is comparison (i.e. multiple data collection points to compare). Assessment of change over time. Done through utilization of:Control groupComparison groupPre- and post-testThe data collected is most often quantitative but may be qualitative given the nature of the program or project.
21 The Four Standards of Evaluation 6/30/2005The Four Standards of EvaluationNo one “right” evaluation. Instead, best choice at each step is options that maximize:Utility: Who needs the info from this evaluation and what info do they need?Feasibility: How much money, time, and effort can we put into this?Propriety: Who needs to be involved in the evaluation to be ethical?Accuracy: What design will lead to accurate information?Successful Grant Writing
22 Step-by-Step Evaluation Design 6/30/2005Step-by-Step Evaluation Design1. Engage stakeholders: Decide who needs to be part of the design and implementation of the evaluation for it to make a difference.2. Describe the program: Draw a “soup to nuts” picture of the program— activities and all intended outcomes.3. Focus the evaluation: Decide whichevaluation questions are the keySuccessful Grant Writing
23 Step – by – Step (continued) 6/30/2005Step – by – Step (continued)Seeds of Steps 1-3 harvested later:4. Gather credible evidence: WriteIndicators, then choose and implementdata collection sources and methods5. Justify conclusions: Review andinterpret data/evidence to determinesuccess or failure6. Use lessons learned: Use evaluationresults in a meaningful way.Successful Grant Writing
24 Designing an Evaluation Plan STEP 1: Engage StakeholdersWho are major stakeholders for our effortsWhere in this model doe they want to see success?Who needs to be engaged upfront to ensure use of results?
25 Group Exercise: Engaging Stakeholders Consider which stakeholders are key to this evaluation of your program or caseConsider what aspects of the program and its evaluation we must attend to, to keep these stakeholders “at the table”
26 Designing an Evaluation Plan STEP 2: Describe the programYou don’t always need a logic model, BUT you ALWAYS need a program descriptionDon’t jump into planning or evaluation without clarity onThe big need your program is to addressThe key target group(s) who need to take actionThe kinds of actions they need to take (your intended outcomes or objectives)Activities needed to meet those outcomesCausal relationships between activities and outcomes
27 Proposal Problems STEP 3: Setting Evaluation Focus Today, Year 1, Year 3, Year 5, where in the model should I be measuring change?If no change, where should I look for problems?
28 Key Domains in Evaluation Focus Implementation (Process)Is program in place as intended?Effectiveness (Outcome)Is program achieving its intended short-, mid-, and/or long term effects/outcomesEfficiencyHow much “product” is produced for given level of inputs/resources?Causal AttributionIs progress on outcomes due to your program?
29 Setting Focus: Some Rules Based on “utility” standardPurpose: Toward what end is the evaluation being conducted?User: Who wants the information and what are they interested in?Use: How will they use the information?
30 Some Evaluation Scenarios Scenario 1: At year 1, other communities want to adopt your model but want to know “what they are in for?”Purpose: Examine program implementationUser: The “other community”Use: To make a determination, based upon your experience whether or not they want to adopt this project
31 Group ExerciseYOU are the community next door that has heard about the program and its progress in its first yearYOU want to try it but wonder what you’re in for. What kinds of things in particular are you looking for data on?