Creating an Evaluation Plan Freya Bradford, Senior Consultant NorthSky Nonprofit Network May 23, 2013

How is the landscape changing?

Learning has never been more important LeadershipAdaptability Program Capacity Sustainability “Anyone who keeps learning stays young.” – Henry Ford TCC Group Sustainability Formula

Phases of Evaluation 1. Create Evaluation Plan 2. Collect & Analyze Data 3. Make Meaning & Adapt Program

Evaluation Plan: A written document that describes how you will assess the success of a program or project.

Evaluation Planning Process Convene a Team Create a Logic Model Prioritize & Define Eval Questions Create a Data Collection Matrix Write & Share the Eval Plan

Convene a team Evaluator/Facilitator (if you have one) Project lead Board member/leadership Key staff person Member of target population Evaluation

Create a logic model

Prioritize & Define Eval Questions “Not everything that counts can be counted, and not everything that can be counted counts.” Measure what matters! Think about uses for the data - Can you identify a decision that could be made as a result of the data? Consider Process & Outcome

Example Questions – NS Coaching Program Process Evaluation Questions 1) Were project activities conducted as planned? Were the outputs achieved? 2) What are the characteristics of participants that engaged in coaching? 3) How satisfied are participants with the coaching that they received? Did it meet expectations? Outcome Evaluation Questions 1) To what extent did coaching help participants meet individual and organizational goals? 2) In what ways did participants and organizations change/grow as a result of coaching?

Create a data collection matrix What Information will be Collected? (Indicators). From whom? (Sources) In what way? (Methods) By whom? (Person Responsible) How will the information be stored and managed? (Data Management)

Create a data collection matrix

Indicators An indicator is piece of data that you will collect to help tell you if you are achieving what you intended Can be Quantitative or Qualitative You may have multiple indicators that you want to collect to measure one part of an Activity or Outcome Common Activity (Process) Indicators FTE/volunteer hours contributed # of partners Amount of products/services delivered #/type of clients served #/type of materials produced/disseminated Timeliness of service provision Quality of services (satisfaction data) Common Outcome Indicators #/% demonstrating increased knowledge/skill #/% demonstrating attitude or behavior change % changes in conditions (longer-term)

Indicators Example Activity of Coaching: Two (2) 45-minute introductory information sessions will be led by coach. They will be promoted through the NS weekly update. Indicators to measure progress: - # of times promoted in weekly update - # clicked through - # of participants at intro sessions - Participant demographics – org. role, tenure in position

Indicators Example Outcome of Coaching: Participants will improve in individual development goal areas (e.g. managing others, self-awareness, self-management, work-life balance). Indicators to measure progress: - % of participants improving by goal area - % of participants meeting and exceeding desired goal - % change in goal achievement scores overall and by goal type

Indicators

Methods Activity tracking Document review Surveys Tests Focus groups Interviews Case studies

Methods – When to Use a Survey Need a little info about a lot of things Have a large number in your target population Numbers are important to your decision making Respondents might not feel comfortable talking about answers in a group Need information quickly

Methods – Tips on Survey Have someone available who can help with survey construction Use a variety of question types – not too many open-ended Consider using an online survey Pilot test survey with group similar to your target Incentives!

Methods – When to Use a Focus Group You have deep or complex issues you want to understand more completely You want to develop some preliminary understanding of the issues surrounding your topics You want to hear people’s deep feelings or insights about your topics Some questions need to be explained in detail or probed to elicit good feedback People need time to ponder questions before responding

Methods – Tips on Focus Group Use a trained facilitator who is as neutral as possible Use a recording device (with permission) and/or a note taker in addition to facilitator Limit groups to 6 – 12 participants Give participants background info before the session Carefully plan questions – 3 – 4 in-depth questions with planned probes if needed Incentives! Send thank you notes

Indicators

Write & Share the Eval Plan Evaluation Plan Outline I.Introduction II.Evaluation Design & Questions III.Evaluation Methods Method Description Indicators measured with method (Repeat for each method) IV.Use of Findings Attachments: Timeline for Evaluation Implementation Logic Model

Resources NorthSky Resource Center This presentation and other free resources Including: University of Wisconsin-Extension, Program Development and Evaluation Unit: United Way Outcome Measurement Resource Network: W.K. Kellogg Foundation Logic Model Handbook: