1. Earmark Grant Evaluation: An Introduction and Overview May 2005 Presented by: Nancy Hewat, Senior Project Manager Public Policy Associates, Inc. 119 Pere Marquette Drive Lansing, Michigan 48912-1231 (517) 485-4477 www.publicpolicy.com

2. Presentation Topics uThe evaluation requirement for earmark grants uEvaluation overview – or – “Where’s the upside?” uPlanning the evaluation uLogic modeling uThe evaluation process for earmark grants Please don’t hold your questions!

3. The Evaluation Requirement

4. Each grantee must … uConduct or commission an evaluation uSubmit evaluation plan uUse the evaluation template uSubmit evaluation report shortly after completion of project activities

5. Evaluation Overview – or – “Where’s the upside?”

6. Evaluation is a mindset … uWe are all evaluators uEvaluation is continuous uEvaluation looks forward, not just backward uInvolves organizational learning uMeans people working together

7. Program evaluation is... uThe systematic collection of information about the subject of the evaluation uUsed to make decisions about organization’s or program’s: 4 Creation 4 Improvement 4 Effectiveness

8. Evaluation allows you to examine... uWhat’s working well uWhat is not uHow to improve There is no bad news, only news!

9. evaluation showing results improve program/ project quality (learning from experience) futurepastpresent Evaluation looks in two directions

10. Evaluation requires comparison... 4of the same group over time pre- and post-tests trends in community-level data 4of two comparable groups l at one point in time l over time 4of your group to a larger group l county compared to state

11. Our Approach: Utilization-Focused Evaluation uFocuses on intended uses and intended users uIs inherently participatory and collaborative by actively involving primary intended users in all aspects of the evaluation uLeads to ongoing, longer-term commitment to using evaluation logic and building a culture of learning in a program or organization

12. Benefits of Evaluation uProgram/organizational improvement uAccountability to funders and others uPlanning uProgram description for stakeholders uPublic relations uFund raising uPolicy decision making Evaluation has lots of upside!

13. Planning the Evaluation

14. Elements of the Evaluation Plan uWho conducts the evaluation? 4Internal or external? 4Experienced or novice? uWhen do they do it? 4Along the way or after the fact? uHow much do they do? 4The level of intensity must fit the project 4Too much diverts resources, too little leaves unanswered questions uWhat exactly do they do? 4Six major steps

15. Evaluation Steps 1. Specify goals2. Establish measures 3. Collect data 4. Analyze data 5. Prepare reports 6. Improve project

16. Step 1: Specify Goals uThinking about goals 4What are you trying to accomplish? 4What would success look like? 4What is the difference between the current state of affairs and what you are trying to create? uExample of a goal statement: “Increase incomes of low-income families in the region through training for entry-level jobs that have career ladders leading to good jobs.”

17. Step 2: Establish Measures uDetermine performance measures 4Must be quantifiable 4Data must be available, reliable, and valid uExamples of measures: Process: Number of trainees Outcome: Skill and credential gains Impact: Wage increases and promotions

18. Step 3: Collect Data uIdentify data sources, such as: 4Administrative records 4Surveys, interviews, focus groups 4Observation uGather data 4Design the instruments and procedures for collection 4Conduct data collection periodically uRecord data 4Organize data 4Create data base 4Verify data Remember the measures!

19. Step 4: Analyze and Interpret Data uSort and sift: organize data for interpretation 4Cross tabs 4Modeling uConduct data analysis to look for: 4Changes over time 4Progress relative to goals or standards 4Differences between groups uTest preliminary interpretation This is the most creative step.

20. Step 5: Prepare Reports uDetermine reporting schedule uReport preliminary findings to key stakeholders and other audiences uGather reactions uIncorporate reactions uFinalize reporting products Different audiences need different types of reports.

21. Step 6: Improve Project uDeliver reporting products internally uFacilitate strategic and operational planning uImprove processes and results A good evaluation will be more valuable to you than to DOL!

22. Logic Modeling

23. Does the project hang together? uAre the expected outcomes realistic? uAre there enough resources? uDo the customers like the product? uDoes the organization have the right skills? Logic models help answer these questions.

24. A Simple Logic Model Things needed to run the project: People, stuff, money, etc. What you do: Market, recruit, design, train, place, etc. Direct results of activities: Training completers, credentials awarded, etc. Changes caused by the project: Jobs, wages, promotions, etc. InputsActivitiesOutputsOutcomes

25. Logic Models Focus on Outcomes Mission Concise statement of purpose Goal Broad statement of desired outcome Objective Measurable statement of an expected outcome over a period of time Performance Measures Ongoing quantitative indicators of objective outcome achievement

26. The Evaluation Process for Earmark Grants

27. Use the DOL Tools u“The Essential Guide for Writing an Earmark Grant Proposal” u“Evaluation Template for Earmark Grantees” (to be provided later)

28. Earmark Grant Evaluation: An Introduction and Overview May 2005 Presented by: Nancy Hewat, Senior Project Administrator Public Policy Associates, Inc. 119 Pere Marquette Drive Lansing, Michigan 48912-1231 (517) 485-4477 www.publicpolicy.com