1 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Collecting Data This is STEP 3 of the five steps of evaluation (see “Planning an Evaluation”).

2 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Test your data collection IQ! Answer the questions on the next slide… and then find the answers on the following slides.

3 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation True or False? (answer each statement with either true or false) a.There is one best way to collect data b.Quantitative methods that collect numbers provide more useful information c.Evaluation data collection involves any established social science research method d.We often collect data from program participants e.We should always collect data from as many participants as possible

4 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation a.FALSE. Unfortunately, there is NOT one best way to collect data. It all depends…upon what you want to know, who the respondents are, your resources and what is culturally appropriate among other things. Think about your choices and weigh the pros and cons of different ways to collect data b.FALSE. Sometimes numbers are more useful; other times, narrative (qualitative data) are more useful. Often times, a mix of quantitative and qualitative data provides the most useful information c.TRUE: any and all social science research methods can be used for collecting evaluation data

5 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation d.TRUE: Yes, we often do collect data from program participants BUT, that doesn’t mean they are the ONLY ones or the best ones. There may be others (parents, volunteers, friends) who might be in a better position to provide the information we need. Or, we might be able to use existing documents, record books, journals, photographs as sources of information. e.FALSE. Sometimes using a sample provides better data.

6 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Use this booklet for help Collecting data involves: 1)Sources - where you will get the information; and 2)Methods - how you will collect /gather the information

7 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation SOURCE of information From where or from whom will you get the information? Existing information – records, reports, program documents, logs, journals People – participants, parents, volunteers, teachers Pictorial records and observations – video or photos, observations of events, artwork

8 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation METHODS of data collection Survey Case Study Interview Observation Group assessment Expert or peer reviews Portfolio reviews Testimonials Tests Photograph, videotape, slides Diaries, journals, logs Document review and analysis Common methods include:

9 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Methods are often thought of as quantitative or qualitative... Quantitative methodsQualitative methods Surveys Questionnaires Focus groups TestsUnstructured interviews Existing databasesUnstructured observations

10 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Quantitative and Qualitative information "Not everything that counts can be counted." 5 (Quantity) Happy (Quality) Kids

11 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Quantitative data collection methods produce numbers. Qualitative data collection methods produce words. Quantitative and qualitative each has its strengths and weaknesses. Quantitative methods are more structured and allow for aggregation and generalization. Qualitative methods are more open and provide for depth and richness.

12 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation What method shall I use? There is no simple answer There is no ONE best method It all depends…

13 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation When choosing methods, consider… The purpose of your evaluation − Will the method allow you to gather information that can be analyzed and presented in a way that will be credible and useful to you and others? The respondents − What is the most appropriate method, considering how the respondents can best be reached, how they might best respond, literacy, cultural considerations, etc.?

14 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Consider… What kind of data your stakeholders will find most credible and useful Resources available. Time, money, and staff to design, implement, and analyze the information. What can you afford? Type of information you need. Numbers, percents, comparisons, stories, examples, etc.

15 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Interruptions to program or participants. Which method is likely to be least intrusive? Advantages and disadvantages of each method..The importance of ensuring cultural appropriateness. Consider…

16 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Often, it is better to use more than one data collection method. Why would this be so?

17 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation When we use several methods we say we are ‘triangulating’. Triangulation is important in evaluation because we want accurate and trustworthy information. Triangulation means the use of multiple sources and methods to gain a better understanding. Each source and each method has inherent biases so using more than one source and/or method provides a more accurate picture.

18 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation How might you mix sources of information in your evaluation? How might you mix data collection methods to evaluate your program?

19 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Mix sources of information For example, you might collect information from program participants AND parents; or from campers and camp leaders. Mix data collection methods For example, you might survey participants AND interview a sample of participants. You might conduct focus group interviews with community service participants AND observe the community service projects. Some ideas

20 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Mixing sources and methods Thinking back on the examples in the previous slide, what different type of information might you get from the different sources and methods? Using multiple sources and/or methods means more time and resources. The choice of data collection method ultimately depends upon the resources you have available.

21 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Instrument to collect the information What will you use to actually collect the data… a recording sheet, a questionnaire, a video or audio tape?

22 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation The term, “instrument”, sounds like we are talking about a dental office, a cockpit or an orchestra. Actually, we use the term “instrument” to mean the tool on which the data is actually recorded: the questionnaire, the recording form, the video or audio tape, for example. If you have selected a survey as your method, you automatically know that you will need a questionnaire. But, if you choose a method such as focus group or interview or observation, think about what you will use for recording the information.

23 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Choices: Timing When Will Data be Collected? –Before and after the program –At one time –At various times during the course of the program –Continuously through the program –Over time - longitudinally

24 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Sample Will you collect information from everyone or use a sample?

25 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation A sample is a portion or a subgroup of a larger group called a population. Sometimes we want to include everyone; other times it is better to collect information from a sample. See the booklet, Sampling Also, see Bill Trochim’s Social Science Research Methods at

26 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Types of sampling strategies: Probability: Why? Generalize to population. Some examples: –Simple random sample –Stratified sample –Cluster sample –Systematic sample Nonprobability: Why? Generalizability not as important. Want to focus on “right cases.” Some examples: –Quota sample –“Purposeful” sample –“Convenience” or “opportunity” sample

27 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Re-test your data collection IQ now! a.There is one best way to collect data b.Quantitative methods that collect numbers provide more useful information c.Evaluation data collection involves any established social science research method d.We often collect data from program participants e.We should always collect data from as many participants as possible True or False?

28 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Reflection time What is one thing you learned (or had reinforced) from going through this presentation that you hope not to forget? Good luck with your data collection efforts!