May 5, 2015 Strategies for Evaluation Data Collection Eric Graig, Ph.D.

Slide 2 Data Collection for Evaluation Session Objectives Participants will understand the factors to consider when selecting data collection methods Participants will become familiar with the data collection methods most often used in evaluation studies Participants will leave with an understanding of the resources Innovation network and LSC are providing to help develop effective data collection tools

Slide 3 Data Collection for Evaluation More Information

Slide 4 Data Collection for Evaluation Starting Points Evaluation questions Evaluation plan

Slide 5 Data Collection for Evaluation Choosing a Data Collection Strategy Process evaluation Outcome evaluation Organizational capacity Budget Expertise Staff time Desired / Required N Experiments Surveys Standardized instruments Interviews Focus groups Review of existing records

Slide 6 Data Collection for Evaluation Integrate Data Collection into Operations The best data collection strategy is one that is seamlessly integrated into you day-to-day operations. Think of your organization as an information system. Where in your workflow can program evaluation information be best captured?

Slide 7 Data Collection for Evaluation Be Systematic Effective evaluation requires a close attention to detail. You needn’t be scientific but you should strive to be systematic. Record keeping is primary. Encourage your left-brain to create forms and protocols. Write everything down.

Slide 8 Data Collection for Evaluation Example 1

Slide 9 Data Collection for Evaluation Example 2

Slide 10 Data Collection for Evaluation When to Use Surveys When your evaluation questions are well defined and the range of possible answers is limited When you need to collect data from a fairly large number of people and you have access to them and can reasonably expect them to cooperate with the survey When you have the ability to process and analyze the relatively large amount of data you will collect

Slide 11 Data Collection for Evaluation Kinds of Survey Questions Knowledge questions Attitude questions Behavior questions Subject characteristics

Slide 12 Data Collection for Evaluation Generating Survey Questions Evaluation Questions Evaluation Sub-questions Operational Questions Individual Survey Items

Slide 13 Data Collection for Evaluation Generating Survey Questions Evaluation Questions Evaluation Sub-questions Operational Questions Individual Survey Items What was learned? How was identity affected? Were students satisfied? Satisfied w/ registration proc. Satisfied with support received Satisfied w/ course component Relevance of content to visits Quality of the instructors Convenience of class schedule Right days Right time Right class length

Slide 14 Data Collection for Evaluation Writing Survey Questions Question formats Multiple choice Ordinal scales Multi-punch questions Number of choices provided Even number for a scale Strive for clarity Use appropriate language Avoid words with multiple meanings Be careful around dates and amounts

Slide 15 Data Collection for Evaluation Creating Neutral Questions Dealing with obviously sensitive topics Dealing with less obviously sensitive topics How often do you attend religious services? 1.Once a week or more 2.Several times a month 3.About once a month 4.Several times a year 5.Never Some people attend religious services once or several times each week. Others attend only several times a year or not at all. Thinking back over the last year, how often would you say you attended religious services?

Slide 16 Data Collection for Evaluation Survey Delivery In-person paper and pencil Mail paper and pencil Internet Telephone

Slide 17 Data Collection for Evaluation Questions

Slide 18 Data Collection for Evaluation When to Use Interviews When you are trying to understand someone else’s subjective experience. When your evaluation questions are more open-ended and the range of possible answers is large. When you don’t need to generalize your findings to a larger group.

Slide 19 Data Collection for Evaluation Interview Questions Avoid questions that lead to ‘motive talk’ Try to use: Could you tell me about that Please tell me more about that How did that work / come about / feel Ask interviewees to provide a narrative account of what happened or how they felt

Slide 20 Data Collection for Evaluation Tips for Interviews Spend time on small talk in order to make the interviewee feel comfortable Use a written set of questions with probes Follow the protocol but remain open to exploring other themes as they arise Record if at all possible

Slide 21 Data Collection for Evaluation When to Use Focus Groups When you are trying to understand subjective experience When your evaluation questions are more open- ended When you sense that respondents may be uncomfortable in a one-on-one interview When your evaluation questions require you to understand a group dynamic

Slide 22 Data Collection for Evaluation Tips for Focus Groups Same as for interviews Plan on 8 to 10 participants and accept that one or two may not show up Get a good mix of participants Learn how to prevent particular participants from dominating the group and how to encourage wall flowers to speak up Consider an introductory exercise to get the people comfortable with one another

Slide 23 Data Collection for Evaluation Questions

Slide 24 Data Collection for Evaluation When to Program Records When you have them and can access them When you can vouch for their accuracy and reliability and when you know they are complete When you can understand them

Slide 25 Data Collection for Evaluation Tips for Analyzing Program Records Very their accuracy Find out what, if anything, is missing Be systematic Plan on making recommendations about how to improve them to make them more useful for evaluation

Slide 26 Data Collection for Evaluation Questions

Slide 27 Data Collection for Evaluation Additional Resources InnoNet website Asynchronous material Synchronous course One-on-one technical assistance