Prioritize & Define Eval Questions “Not everything that counts can be counted, and not everything that can be counted counts.” Measure what matters! Think about uses for the data - Can you identify a decision that could be made as a result of the data? Consider Process & Outcome
Example Questions – NS Coaching Program Process Evaluation Questions 1) Were project activities conducted as planned? Were the outputs achieved? 2) What are the characteristics of participants that engaged in coaching? 3) How satisfied are participants with the coaching that they received? Did it meet expectations? Outcome Evaluation Questions 1) To what extent did coaching help participants meet individual and organizational goals? 2) In what ways did participants and organizations change/grow as a result of coaching?
Create a data collection matrix What Information will be Collected? (Indicators). From whom? (Sources) In what way? (Methods) By whom? (Person Responsible) How will the information be stored and managed? (Data Management)
Indicators An indicator is piece of data that you will collect to help tell you if you are achieving what you intended Can be Quantitative or Qualitative You may have multiple indicators that you want to collect to measure one part of an Activity or Outcome Common Activity (Process) Indicators FTE/volunteer hours contributed # of partners Amount of products/services delivered #/type of clients served #/type of materials produced/disseminated Timeliness of service provision Quality of services (satisfaction data) Common Outcome Indicators #/% demonstrating increased knowledge/skill #/% demonstrating attitude or behavior change % changes in conditions (longer-term)
Indicators Example Activity of Coaching: Two (2) 45-minute introductory information sessions will be led by coach. They will be promoted through the NS weekly update. Indicators to measure progress: - # of times promoted in weekly update - # clicked through - # of participants at intro sessions - Participant demographics – org. role, tenure in position
Indicators Example Outcome of Coaching: Participants will improve in individual development goal areas (e.g. managing others, self-awareness, self-management, work-life balance). Indicators to measure progress: - % of participants improving by goal area - % of participants meeting and exceeding desired goal - % change in goal achievement scores overall and by goal type
Methods Activity tracking Document review Surveys Tests Focus groups Interviews Case studies
Methods – When to Use a Survey Need a little info about a lot of things Have a large number in your target population Numbers are important to your decision making Respondents might not feel comfortable talking about answers in a group Need information quickly
Methods – Tips on Survey Have someone available who can help with survey construction Use a variety of question types – not too many open-ended Consider using an online survey Pilot test survey with group similar to your target Incentives!
Methods – When to Use a Focus Group You have deep or complex issues you want to understand more completely You want to develop some preliminary understanding of the issues surrounding your topics You want to hear people’s deep feelings or insights about your topics Some questions need to be explained in detail or probed to elicit good feedback People need time to ponder questions before responding
Methods – Tips on Focus Group Use a trained facilitator who is as neutral as possible Use a recording device (with permission) and/or a note taker in addition to facilitator Limit groups to 6 – 12 participants Give participants background info before the session Carefully plan questions – 3 – 4 in-depth questions with planned probes if needed Incentives! Send thank you notes
Write & Share the Eval Plan Evaluation Plan Outline I.Introduction II.Evaluation Design & Questions III.Evaluation Methods Method Description Indicators measured with method (Repeat for each method) IV.Use of Findings Attachments: Timeline for Evaluation Implementation Logic Model
Resources NorthSky Resource Center This presentation and other free resources http://www.northskynonprofitnetwork.org/resource-center Including: University of Wisconsin-Extension, Program Development and Evaluation Unit: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.htmlhttp://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html United Way Outcome Measurement Resource Network: http://www.liveunited.org/Outcomes/Library/pgmomres.cfm W.K. Kellogg Foundation Logic Model Handbook: http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf